ICRR 12584 Report Number : ICRR12584 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 04/04/2007 PROJ ID : P040196 Appraisal Actual Project Name : Sumatra Basic US$M ): Project Costs (US$M): 84.3 98 Education Project Country : Indonesia Loan/ US$M ): Loan /Credit (US$M): 74.6 73.9 Sector Board : ED US$M): Cofinancing (US$M ): Sector (s): Secondary education (27%) Tertiary education (26%) Primary education (26%) Sub-national government administration (12%) Central government administration (9%) Theme (s): Education for all (29% - P) Rural services and infrastructure (29% - P) Administrative and civil service reform (14% - S) Decentralization (14% - S) Participation and civic engagement (14% - S) L/C Number : C3189; L4456 Board Approval Date : 04/08/1999 Partners involved : Closing Date : 04/30/2006 04/30/2006 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Roy Gilbert Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The project aimed to (a) To mitigate the effect of the economic crisis on basic education enrollments and operation of primary and junior secondary schools by : (i) Maintaining primary and junior secondary enrollment rates and transition rates from primary to junior secondary for the poor; (ii) preventing quality from deteriorating further by ensuring that school can meet essential operational maintenance costs; and (iii) realizing efficiency gains where possible; and (b) to support recovery and return to the medium -term education strategy of: (i) improving quality of basic education (grades 1 – 9); and (ii) decentralizing educational planning, management and implementation effectively to districts and below . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) Crisis relief , that included scholarships, school grants, and training for school committees (US$27.6m at appraisal, US$17.06m actual). (b) Institutional development that included (i) strengthening education management through school mapping and a five-year integrated plan at all levels covering school location, teacher training / support, provision of books / teaching aids, promotion of community participation; (ii) organization and staffing assessment of all levels; (iii) capacity enhancement through staff training, student assessments and project impact evaluation, and office equipment . Management was to be strengthened through regular reporting / monitoring, technical audits, project management training, office equipment; and preparation activities for teacher policy reforms (US$9.8m at appraisal, US$4.77m actual). (c) Quality enhancement that included: (i) consolidation of small schools into larger entities; (ii) school rehabilitation and maintenance; (iii) provision of books for supplementary reading etc .; (iv) training for trainers, teachers and library managers, school principals supervisors, and community leaders; employment of 800 contract teachers and grants for private schools (US$38m at appraisal, US$12.69m actual). Following the 2000 decentralization law, the project was restructured in 2002. Some components were re-grouped for better focus at the district level . Components after restructuring were : (a) District education program for (i) school improvement program, through rehabilitation and construction grants, and quality improvement grants (for learning materials, teacher training etc .); (ii) contract teachers; (iii) updating of school maps; (iv) . private school grants; and (v) support services, including district three -year plans, training for school committees, and monitoring and evaluation by districts reforms (US$00m at appraisal, US$48.1m actual). (b) Provincial support for district education program through (i) upgrading for education professionals; and (ii) support for district programs, including training for district managers on project activities, technical assistance for project implementers, and (iii) in-country study tours (US$00m at appraisal, US$7.96m actual). (c) Central support for district education program through (i) support to district education program activities through training and in-country study tours; and (ii) district-level teacher management and teacher support services (US$00m at appraisal, US$3.78m actual). (d) Crisis relief through (i) scholarships; (ii). school block grants; and (iii) training (US$27m at appraisal, US$17.06m actual-as above). (e) Project management (US$6.4m at appraisal, US$3.13m actual). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project consisted of a loan of US$ 47.9m and an IDA credit of US$15.9m equivalent. The project was completed on time despite a frequent lack of counterpart funds . Approximately US$0.7m were canceled. 3. Relevance of Objectives & Design: This project supported specific sector -related goals of the government (Indonesia, Short Education Strategy Finance Note) and the 2001 Country Assistance Strategy (No. 18963 IND, paras. 36 and 49). These included short-term goals to reinforce social safety nets through the delivery of improved basic education services to the poor, and in the medium term, the strengthening of human resource development in rural areas by decentralizing key decision making to the lowest efficient level (district and community). Also associated with this goal, the project sought to improve institutional development in basic education, and the internal and external efficiency of basic education through wide scale quality improvements in those areas . The design allowed for significant integration and coordination of consolidated activities in primary and secondary education . 4. Achievement of Objectives (Efficacy): (a) Mitigating the effect of the economic crisis (substantial). Despite the economic crisis, net enrollment rates for the target provinces were maintained or increased in basic education during the implementation of the project . (Target was no loss.) For example during the period 1999-2006, primary net enrolment changed from 88% to 90% in north Sumatra, from 85% to 92% in Rau, and from 88% to 92% in Benkulu. As a result of the school grants program, the transition rates from primary to junior secondary school did not decline significantly during the life of the project . Secondary net enrollment rates during 1999-2006 in north Sumatra changed from 74% to 73%, in Rau from 67% to 72%, and in Benkulu from 65% to 69%. For each year of the program, all of schools that received school grants remained open. The project's scholarships were targeted to poor students, and a study showed that their enrollments were largely maintained. (Studies not cited in the ICR, see Indonesia Junior Secondary Education Project PPAR, 2006 for details). (b) Support recovery and return to the medium -term education strategy of improving quality and decentralization (substantial ). The project undertook school mapping, with 90% of district plans showing evidence of use; rehabilitation of 893 primary schools (target exceeded by 6%),and construction of two new junior secondary schools in remote areas. Grants to schools aimed at improving quality enabled 2327 schools to obtain books and teaching aids, and 90% of schools showed increased use of teaching aids . Schools built by communities through block grants cost about 25% less and included extra amenities for students at no extra cost . Employment of 1091 contract teachers has allowed districts to deploy teachers to fill gaps in remote schools; 92% of head teachers were satisfied with their performance. Teacher certification for madrasah teachers has enabled them to improve their qualifications (86% met certification requirements; program did not exist before the project ). Decentralization processes continued . Each district prepared a three -year plan reflecting their priorities in budget allocations . School mapping was used as a budgeting and planning tool to allocate resources and take care of schools that have been in poor condition . Decentralization of decision-making down to the community level was significant and empowered communities (a) keep schools open longer to serve both children and adults; (b) expand the services to include school libraries; (c) establish school beautification and landscaping programs for their schools; (e) establish volunteer programs to support schools and community development; and (f) accept contractual responsibility for school upgrading and establishment using local workers . 5. Efficiency (not applicable to DPLs): The planning and budgeting activities required and completed under this project permitted efficiency gains in the use of funds at the district level . Use of contract teachers over fully paid and permanent teachers also made it possible to provide education at a lower cost . Schools built by communities through block grants cost about 25% less and included extra amenities for students at no extra cost . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project met all of its development objectives . Basic education in low-income areas was protected from the financial crisis; furthermore quality and institutional development were adapted to the needs of the district -based decentralization policy. Thus it supported recovery and a return to the medium -term education strategy of education. The decentralization of educational planning to district level and below took place during the project . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The project successfully responded to a financial crisis, and it is possible that the government will respond satisfactorily to other crises . Despite several financial management challenges during the life of the project, districts have been able to receive the tools to manage the provision of education effectively . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: The Bank was flexible and able to offer viable solutions to the financial and administrative challenges that faced the project. Quality at entry was considered satisfactory . It also was vigilant and cooperative with the government on procurement issues that arose with respect to civil works . at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Overall performance was satisfactory . The government showed commitment and after some delays made funds available. There were initial disruptions when staffing was not available to effectively implement the project. Overall performance was satisfactory . The government showed commitment and after some delays made funds available. There were initial disruptions when staffing was not available to effectively implement the project . The project experienced problems with procurement, disbursement, but the implementation unit worked in close collaboration with the Bank to make necessary changes in the project and bring the project back on schedule . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: The PAD did not include a clear monitoring design and assessment criteria . After the second year of the project, however, data collection started . Subsequently measurable key performance indicators were updated and used during supervision mission. In the last year of the project, consultants visited project sites and collected quantitative and qualitative data. However, the project never had an evaluation design that would permit a study of the impact of project inputs on student outcomes . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The project experienced management and procurement problems, which were not always effectively mitigated . These were to some extent due to the economic crisis . However, province- and district-level management posed difficulties and recurrent concerns about weak procurement, lack of staffing, and inadequate funding . There was also a lack of clarity in understanding the changing roles of government under decentralization . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: - Large-scale decentralization should proceed cautiously . Incentives should be carefully thought through, decided upon, resourced and made consistently available, before community participation and the establishment of decentralized education programs begin . Otherwise, staff, management, and planning mechanisms may not be ready to accommodate the decentralization needs . - Communities in many countries are able to assume some responsibility and to construct and equip schools through the help of block grants . However, training and follow-up are needed for this level of involvement to occur . - It is possible to mitigate the effects of an economic crisis through targeted interventions at vulnerable students attending critical stages of their education . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The original version of the ICR received by IEG through the system had missing text and inconsistencies in key figures. The region subsequently provided the corrected version which it confirmed is the one being printed and publicly disclosed. On the basis of the latter version, ICR quality is rated satisfactory overall . However, the ICR follows the presentation used in the PAD of a generic main text and province -specific annexes, which makes for rather awkward reading. While the ICR annexes provide Sumatra -specific data on outputs and KPIs, the main text could have usefully elaborated on these . a.Quality of ICR Rating : Satisfactory