ICRR 12751 Report Number : ICRR12751 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/21/2007 PROJ ID : P072647 Appraisal Actual Project Name : Tp-fundamental US$M ): Project Costs (US$M): 14.10 20.6 School Quality Project Country : Timor-Leste Loan/ Loan /Credit (US$M ): US$M): 20.6 20.4 Sector Board : ED US$M ): Cofinancing (US$M): Sector (s): Primary education (44%) Secondary education (44%) Central government administration (12%) Theme (s): Education for the knowledge economy (33% - P) Education for all (33% - P) Participation and civic engagement (17% - S) Conflict prevention and post-conflict reconstruction (17% - S) L/C Number : Board Approval Date : 09/28/2001 Partners involved : Closing Date : 11/30/2003 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Ridley Nelson Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The project aimed to: (a) support the Government's efforts to maintain the existing level of primary education enrollment and to restore junior secondary enrollments to the pre -1999 levels with the possibility of increase due to return of refugees and growth of school -age population; and (b) continue to recover the quality of primary and junior secondary education by rehabilitating physical facilities and by providing textbooks and instructional materials . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) Rebuilding school infrastructure (US$9.8m at appraisal, US$15.77m actual) for (i) construction of about five escolas basicas (integrated primary and junior secondary schools ) at the district level; (ii) upgrading of about nine selected primary and junior secondary schools to the standard of escolas basicas; and (iii) upgrading of about 65 selected primary schools to the fundamental quality standard that included furniture, water, and sanitation . The project was revised at midterm review to include only 35 primary schools and use a single standard school design. Providing teaching -learning materials (US$1.3m at appraisal, US$1.24m actual) for primary and junior secondary (b)Providing schools that included textbooks, workbooks, and teachers ’ guides for language learning, kits for science and mathematics, computers, and software . (c) Social mobilization and communications (US$0.3m at appraisal, US$0.2m actual), including production and dissemination of promotional materials, delivery of social messages, development and carrying out of training programs, information campaigns, and monitoring, feedback -gathering, and evaluation. (d) Support for policy development (US$0.2m at appraisal, US$0.97m actual), including technical and empirical studies on education issues and policy options, data management , and dissemination of knowledge regarding the development of the education sector . (e) Management and implementation support (US$10m at appraisal, US$2.42m actual), including support to management, operational, and administrative activities of the project management unit (PMU). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Supplemental Grant Agreement of US$6.7 million was provided and was used for rebuilding schools . The project was extended three times due to continuing civil strife and closed on 12/31/2006. About US$0.2m were canceled (ICR does not give exact amount ). 3. Relevance of Objectives & Design: No Country Assistance Strategy had been formulated at the time of appraisal . However, the project supported specific sector-related goals of the United Nations Transitional Administration in East Timor and the government . It also supported three of the five objectives set by the Transitional Support Strategy of the World Bank Group for Timor-Leste (November 3, 2000). The goal of improving the quality and access to primary and junior secondary education continued to be relevant after project completion . However, the project goals included quality only in terms of textbooks and inputs rather than teacher training and systemic preparation for a new language of instruction . Furthermore, a two-year project period was unrealistic for delivering the above inputs . 4. Achievement of Objectives (Efficacy): (a) Supporting government efforts to maintain the existing level of primary education enrollment and to restore junior secondary enrollments to the pre -1999 levels (satisfactory). During appraisal, enrollment before independence had been estimated as 95% at primary level (185,000 students) and 60% at the junior secondary level (30,000 students. (A more detailed analysis in 2006 showed much lower figures for the 1998/99 school year; the gross enrollment rate had been 89 % and 44% for secondary while net rates were 51% and 24% respectively). Under both scenarios, enrollments were maintained and had actually increased . For the 2004/05 school year, the gross enrollment rate was estimated as 119% (to include overage children) at the primary and 59% (41,516 students) at the secondary level. (net enrollment was 80% for primary and 30% for secondary. (b) Recovering the quality of primary and junior secondary education by rehabilitating physical facilities and by providing textbooks and instructional materials (modest). . The evidence provided is largely related to physical outputs rather than the outcome of recovery of educational quality. However, the following are the main achievements : - Partly due to continuing civil strife, the number of civil works was reduced . The project completed 102 schools of the revised target of 123 schools; 13 were completed from other funding sources and the MoEC cancelled plans for 15 proposed project schools and transferred their construction expenses to the national budget . It is not clear whether these were built and in operation . 102 school councils were organized (target was 78) to increase accountability . - The project financed 55,000 sets of Portuguese textbooks and teachers guides for grades 4 and 5. All schools received textbooks in Portuguese (target was 80%). An evaluation showed uneven distribution and little teacher training for their use. - Starting in 2004, a publication in the Tetum language called Lafaek was distributed widely and really served as the only material available in rural areas . It reached 105,000 children in grades 4-9. A junior magazine, Lafaek Ki’ik reached 155,000 school children in classes 1-3 and pre-school. The teacher’s guide - reached 7,500 teachers in primary and junior secondary. These were in fact the teaching materials students had available . A 2006 survey by USAID showed that 75 percent of children and their parents were aware of and understood clearly the subjects that are feature in Lafaek publications. - 8 policy studies were produced that made a significant contribution to education policy development . Of particular importance was the 2004 National Education Congress that was attended by 600 education stakeholders. - a primary school achievement assessment study was financed in 2003 (test data are not provided in the ICR ). 5. Efficiency (not applicable to DPLs): There are limited data from which to judge efficiency in the project . School construction took longer than expected and initially used different designs that were standardized after the midterm review . The introduction of the Portuguese language produced inefficiencies in the system that have not been calculated . On the other hand the Lafaek publication was disseminated and used efficiently in rural and urban areas, given that children did not know Portuguese to read the official textbooks . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project achieved the goal of maintaining enrollments to the levels before independence . However, there is no evidence that it achieved the second goal, " continue recovery" in the quality of primary and junior secondary education. (The earlier level of quality was unknown and no targets were given along the recovery continuum .) The project apparently failed to meet a target related to civil works and also to quality of education, requiring scalebacks after the midterm review. Textbook distribution was uneven, making it difficult for students to learn Portuguese; science materials were deleted from the project, and teachers were not sufficiently trained to use a language of instruction they did not know . It seems the production of the Lafaek magazine (distribution financed through CARE ) was the most important contribution to learning outcomes in the project . Some evidence of baseline achievement and potential progress could have been provided in the ICR, but the scores of the achievement test given in 2003, still three years prior to project closing, are not reported . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The educational system may not be producing sufficient levels of learning for students who enroll in schools . Lack of a viable language of instruction, limited teacher training, and uneven textbook distribution may result in continuing low achievement. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Project design at entry was unsatisfactory, partly because the Bank questioned the government language policies (see issues section below). Dialogue was held with the government, but it did not adequately translate into an ajustment of the project design . Thus, the project ultimately provided large quantities of books in Portuguese and none in Tetum. Also the Bank appraised a project to deliver many and difficult inputs in just a two-year period. Subsequently supervision was regular, but the Bank could not effectively help the government overcome the many internal obstacles that slowed down the work . at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The project faced considerable difficulties, partly because it was implemented during a time of ongoing civil strife . During the first 2 years of the project there was very little staff monitoring or accountability for performance . The national consultants hired by the project showed limited commitment . As a result, little was accomplished during the first half of the project. Eventually implementation improved, and (according to the ICR, p. 31) the team implementing the project received an award by the World Bank president for excellence in procurement . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: During appraisal, performance indicators were prepared to support M&E, but there was no evaluation design in place to estimate the effect the project would have on the attainment of objectives . A primary school achievement assessment study was carried out in 2003, but the ICR does not report results . The ICR mentions that data were used for decisionmaking but does not present concrete evidence or examples . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The donors had serious concerns about the viability of the government's choice of Portuguese as a language of instruction, given that almost all people in the country speak a single language (Tetum) that is phonetically written and usable as a language of instruction with updating (as was done in Bahasa Indonesia ). Nevertheless, the issue was not fully discussed, and the Bank went along with government policies to promote a language that has more complex spelling and that few students or teachers know . The Bank did not finance textbooks in Tetum but committed to financing textbooks in Portuguese neither teachers nor students could read . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The objective to improve access was Unsatisfactory achieved, but the quality-oriented objective was not; textbooks were printed but not distributed to students as expected, and teachers were not suffiently trained to use them in Portuguese, so the provision of actual instruction to students was compromised. Achievement tests were given in 2003, but the results are not presented in the ICR. Risk to Development Negligible to Low Significant Teaching students in a language that Outcome : neither they nor the teachers know presents risks producing cohorts of children without much education . The limited achievement of various targets also raises questions about government's ability to continue providing education in the future . Bank Performance : Satisfactory Moderately Project design at entry was Satisfactory unsatisfactory and M&E design, implementation and utilization is rated neglible. The Bank seriously questioned the government language policies (see issues section below) and dialogue was held with the government, but this did not adequately translate into an adequate response in terms of project design. For example, the project provided large quantities of books in Portuguese and none in Tetum. The Bank appraised a project to meet ambitious quality objectives and to deliver many and difficult inputs in just a two-year period which not surprisingly had to be stretched to 5 years. Subsequently supervision was regular, but the Bank could not effectively help the government overcome many of the internal obstacles that slowed down the work . Borrower Performance : Satisfactory Moderately During the first half of the project, staff Satisfactory and consultants had limited accountability and interest in carrying out project activities. The situation improved in the second half, but important targets were missed or scaled back. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Post-conflict reconstruction projects may face obstacles in hiring qualified and competent staff and experience problems in interacting with government officials who are not always well qualified . Despite the Bank's best efforts, their achievements may be limited, and expectations should be suitably modest; Language of instruction is a crucial choice that strongly affects students' ability to perform in school . Prior examples of introducing a language little known by teachers and students (e.g. in Madagascar) have resulted in years of poor performance. This is particularly the case when a language uses irregular orthography, as does Portuguese. Governments must be advised to proceed cautiously and to consider using the national lingua franca as a language of instruction, even though it is not an internationally used language; and Large numbers of small civil works are carried out most efficiently when there are standard designs to be implemented rather than individual designs . Means must be found to reconcile community requests and needs for efficiency. 14. Assessment Recommended? Yes No Why? The language issue warrants more exploration . 15. Comments on Quality of ICR: The ICR explains various project events, but it misses some important analysis and information warranting an unsatisfactory rating: It does not attempt to clarify the quality -oriented objective (b) whose "continued recovery" goal seems unevaluable, or discuss why some activities targeted to this objective (school construction, studies ) seem insufficient or inconsistent with its aim . It does not present the scores of the 2003 achievement test, that would have clarified the achievement of the all-important quality objective or at least provided baseline measurements; It discusses in the text the higher pre -independence enrollment rates used in the appraisal but only mentions in the data annex that there were lower and more realistic enrollment rates; and It offers a limited understanding of why the government could not carry out the project . For example, para 6.4 mentions ''some setbacks..such as project management and design problems ". a.Quality of ICR Rating : Unsatisfactory