ICRR 13117 Report Number : ICRR13117 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 04/14/2010 PROJ ID : P082646 Appraisal Actual Project Name : Community School US$M ): Project Costs (US$M): 5.18 5.69 Support Project Country : Nepal Loan/ US$M ): Loan /Credit (US$M): 5 5.36 Sector Board : ED US$M): Cofinancing (US$M ): Sector (s): General education sector (100%) Theme (s): Education for all (100% - P) L/C Number : C3808 Board Approval Date : 05/12/2003 Partners involved : Closing Date : 09/30/2006 09/30/2008 Evaluator : Panel Reviewer : Group Manager : Group : Helen Abadzi Martha Ainsworth IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA), the objective of the Community School Support Project (CSSP) was to develop a knowledge base needed for successful transfer of responsibility for management of government-funded community schools to local communities . According to the Project Appraisal Document , the objective of the project was to "assist the government to gain critical experience for successfully up-scaling the transfer of management of government-funded schools to communities" (p. 2). PAD Annex 1, p. 16 lists the DCA objective as a “follow-on development objective� and includes a new “project development objective :� To test whether community management of schools can better contribute to enhance (a) participation rates, (b) quality and efficiency, and (c) accountability. The ICR review uses the objective as defined in the Development Credit Agreement as well as the objective mentioned on p. 16 of the PAD. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (a) School grants (US$3.66 million at appraisal, US$4.9 million final) for (i) incentive grants for committees taking over management of government-funded schools; (ii) block grants tied to performance for post -primary schools transferred to community management; (iii) performance grants for improving access to primary education and promotion rates; (iv) supplementary grants to community schools for expanding enrollment to cover additional costs; and (v) other grants for support to marginalized communities, and piloting of innovative approaches . (b) Scholarships (US$0.44 million at appraisal, US$0.13 million final) for scholarships to poor out -of-school children of disadvantaged families (Rs. 500-1000 for the first child in primary and secondary schools and Rs . 250 for subsequent children.) (b) Capacity building (US$0.42 million at appraisal, US$0.3 million final) for (i) assistance to communities for managing schools; (ii) need-based training for teachers and committee members; (iii) orientation for civil servants, local government officials and resource persons; and (iv) policy communications. (d) Monitoring and Evaluation (US$0.48 million at appraisal, US$0.10 million final) for a well-established and smoothly operating M&E system that could support periodic improvements in the experimental design of the LIL . In 2005, a recipient-executed grant was obtained (Policy and Human Resources Development - PHRD - for US$699,600, (of which $562,643 was ultimately disbursed) for: (a) a community awareness and social mobilization, (b) monitoring system for school enrollments, retention and completion, (c) a quality monitoring system for decentralized schools, (d) study on the role of community characteristics on project take -up and recommendations for strengthening of capacities, (e) best-practice handbook and network peer -to-peer support activities, and (f) a project implementation plan. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The credit closed on September 30, 2008 after two extensions totaling 27 months, and disbursed fully. The project was a Learning and Innovation Loan (LIL). A LIL is supposed to focuse on experimentation, learning and piloting in search of possible developmental solutions, prior to potential larger -scale operations. Clarity of the learning objectives and the system to monitor and evaluate them are vital to the purpose of the LIL (Operations Policy and Country Services guidelines,Operations Manual, dated 1998). 3. Relevance of Objectives & Design: The relevance of objective (s) for this project is rated substantial . Community management reflects the government strategy for improved services in several sectors . The 2003 Country Assistance Strategy (CAS) and the first Poverty Reduction Strategy Paper (PRSP) considered community management of schools as the primary means of improving quality of education. Therefore a knowledge base would be important for improving the the performance of community schools. While all schools in Nepal are, on paper, community managed, the extent to which communities were actively engaged in school management was variable . This LIL was to test whether offering a financial incentive to schools to formally transfer to community management enhances participation, quality, efficiency, and accountability to citizens, relative to schools that do not receive that incentive for formal transfer . The relevance of project design is rated negligible . According to Bank guidelines, a LIL should last about 3.5 years and focus “on experimentation, learning and piloting in search of possible developmental solutions, prior to potential larger-scale operations.� There was no evaluation design within the project design document that would have permitted an assessment of the impact of the formal transfer of schools to communities, taking into account the self-selection of communities to be transferred . The formal transfer of schools to community management was not tested against a control group . The design was not guided by analytical work in Nepal on the performance of existing community managed schools or by ex -ante analysis of key stakeholders, like teachers . No clear rationale was given for the grant amounts chosen . Given these design shortcomings, the LIL could not evaluate the effectiveness of formal transfer as a means to improve the access, efficiency, quality, or accountability of education in Nepal . The overall rating for relevance is modest . 4. Achievement of Objectives (Efficacy): Develop a knowledge base needed for successful transfer of responsibility for management of government -funded community schools to local communities (modest ) • The project and the PHRD grant financed innovative activities . These included NGO advocacy to children ’s guardians about their roles in school management; peer -to-peer networks, internal study visits to effectively managed schools, and a best-practice handbook. • The project financed a menu of grants as incentives for committees to accept a formal transfer of their schools to community management. A total of 2,035 parental committees agreed to formally take over school management (The target was to test the intervention in 1500 schools). These schools got an incentive grant of about US$1200 and became eligible for additional grants, including 800 matching grants, 4,914 booster scholarships for poor children, and school -based training for 249 teachers. The ICR, p. 31, states that the CSSP could not ensure continuity to the performance grants, but does not state the reasons . • The initial goal was to test the formal transfer to community management in 1,500 schools. Annex 2 of the ICR shows that target had been met by the FY 2004/2005 school year. Yet, there is no explanation as to why additional schools were added, while the capacity building, scholarship, and M&E elements for the original 1,500 schools were not undertaken . The emphasis of the ICR and the project as implemented seems to have been on the number of schools enlisted rather than on whether the formal transfer to community management improved participation, quality, efficiency, or accountability . • Central and District Steering Committees and Project Advisory and Monitoring Committees "were not created and, if created, were not operational �. “As a result, the proposed technical support for project and school management as well as instructional improvement did not materialize for the majority of the schools and may have hindered progress in instructional transformation .� (ICR, p. 3). Despite various qualitative studies, longitudinal surveys of schools that were formally transferred were not conducted. Data were collected in 2004 and 2007 and were abandoned for reasons that were not explained in the ICR. According to the ICR (p. 20), a qualitative impact evaluation of the project was carried out in 2007, but the results were not provided in the ICR . Overall, the ICR provided no evidence to show how effective the financed activities were in improving Test whether community management of schools can better contribute to enhance : (a ( a) participation rates, (b) quality and efficiency, and (c) accountability (negligible ). The ICR provided evidence only of increased participation rates in about 30 CSSP schools that received "intensive support" (pp. iii, 10, 27, Annex A1, p. 19). The report did not compare these increases to non -participating schools or to schools receiving merely 'incentives' . The data presented show a large decrease in out -of-school children over a single year, between 2005 and 2006, but the ICR offered no evidence that community management rather than other factors were responsible for this change . There is no discussion, for example, of how the end of the internal insurgency in 2006 might have affected enrollment rates all over Nepal, including in the CSSP schools . No specific project output is flagged in 2005 that could have accounted for this sudden, large drop in children out of school in a single year. The project’s evaluation design involved baseline and endline surveys, comparing in the cross -section the results in formally transferred schools and those that had not been formally transferred . However, this strategy does not take into account the fact that communities self -select into the formal transfer group, and those that elect the formal transfers may differ in fundamental ways that also influence the schooling outcomes . Because the evaluation design did not attempt to control for the self -selection of communities into the formal transfer group, the LIL ultimately could not answer the question as to whether formally transferred schools have higher participation rates, quality, efficiency, or accountability, and whether any changes were due to improved management . Despite the information collected through various studies, the ICR did not provide evidence of changes in quality, efficiency (dropout and repetition rates), or accountability associated with the CSSP schools in general, nor did it report trends for teacher absenteeism, which was a major reason for adopting stronger community management . The Borrower’s contribution to the ICR noted that the project placed greater emphasis on increasing the number of community-managed schools than it did on quality improvement (ICR, p. 31). As a result of these shortcomings, the main hypothesis of the project could not be tested, and little was learned about whether formal transfer of schools to community management could enhance the above variables . 5. Efficiency (not applicable to DPLs): The project expended $5 million to learn whether the formal transfer of schools to community management raised schooling outcomes and accountability relative to schools that were not formally transferred, but it did not finance an evaluation that could answer the question . Instead, many other studies were financed, most not comparable, that could not address the project objective, and resources were used to transfer more schools than was planned, even though the efficacy of the intervention had not been demonstrated . Efficiency is therefore rated negligible ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The outcome of the Community School Support Project was unsatisfactory based on modest relevance and efficacy and negligible efficiency . The Bank did not design or supervise under the project a robust project evaluation that would establish cause-effect relationships for this LIL . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Risk to development outcome is rated significant , because the formal transfer to community management has been scaled up to the rest of the country without evidence of its effectiveness in improving participation, quality, efficiency, and accountability . Further, the impact of the formal transfer to community management on the equity of educational outcomes has not been assessed, and there is no assurance of sustained community involvement in transferred schools, particularly in poorer areas . The transferred schools supported by the project were to be managed by committees whose members ’ tenure would be limited to three years; inevitably other community members may succeed them, whose views and dedication are not ensured . Finally, it is unclear how closely and who will monitor committee actions and ensure good governance and fair treatment towards teachers and students . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: The quality at entry of the project was highly unsatisfactory because no robust evaluation design had been developed for the project; there was no risk assessment with respect to the Maoist insurgency or the teachers ’ resistance, objectives were poorly articulated, and the deficiencies in the current system had not been analyzed . The PAD did not explore the possibility that community management might have an adverse impact on equity . Also the envisaged institutional framework needed for implementation, such as a four -committee structure discussed in the PAD did not exist during appraisal and ultimately failed to materialize . The only risk anticipated was the inability of communities to mobilize additional resources (PAD p. 14). The Bank did not consult sufficiently with donors or with prominent stakeholders on project design . Thus, implementation suffered from strong political resistance from teacher groups and educators . Supervision of the CSSP is rated moderately unsatisfactory . Teams of Bank staff and consultants regularly visited schools during supervision missions . A parents' organization (the Community Schools National Network ) was contracted to visit schools and collect data to be used for supervision missions, but some project documents express concerns regarding the reliability of these data . Most critically, the supervision team did not attempt to correct the deficiencies in evaluation design that resulted in the failure of the project to meet its objectives, while launching many non-comparable studies on different samples of schools . Overall, Bank performance is rated unsatisfactory . at -Entry :Highly Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Unsatisfactory 9. Assessment of Borrower Performance: Borrower performance is rated moderately satisfactory . The government was supportive through policies and legislation, and government performance is rated satisfactory . The implementing agency performance is rated moderately satisfactory . For much of the implementation period the project was administered by the Department of Education without specially assigned staff, and administrative arrangements fell short of the scheme envisaged in the PAD. Audits and financial monitoring reports were often late . Given the project design faults, it is unclear to what extent the government intended to learn more about the capacities of communities to manage schools from the operation, as opposed to the ability of schools to absorb cash grants . a. Government Performance :Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design (rated negligible).. The ICR notes that “The need for an independent evaluation was evident since the beginning of the project� (p. 7), yet no clear evaluation design was proposed in the project PAD for this LIL that was capable of evaluating the effectiveness of the formal transfer of schools to community management . There was no discussion of how a counterfactual or control group would be established, or how to take into account the self-selection bias of communities that agreed to participate . Instead, the project simply had a set of indicators in the annex that were to be tracked in the CSSP -financed schools. Implementation (rated modest). The Research Support Group envisaged in the PAD was not created . Documents show that data from a purposive sample of 80 schools were collected in 2004 and then abandoned. Multiple other studies were carried out with trust funds (TF055146), but the data were not comparable and did not provide generalizeable conclusions . The project monitoring reported in the ICR (p. 6) only covered only 30 non-randomly selected “intensive support schools � (those that received both incentive and matching grants ). Though three rounds of household surveys were undertaken, there was no control group . Sampling methods, and thus the representativeness of the studies, were unclear . a significant problem given Nepal’s multiple and variable ecological zones. Overall, multiple studies were conducted through the PHRD grant and the project itself . The impact evaluation on learning outcomes that was the focus of the US$ 699,000 PHRD grant was not delivered at the end of the project. The many disparate studies conducted raise the concern that a considerable amount of money was spent on monitoring and evaluation activities without obtaining interpretable findings . Use of data (rated negligible). There was no definitive answer to the question posed by the project to guide policy, . There has been no summary of the multiple studies that were commissioned . The ICR and trust fund monitoring reports stated that findings would be available in the first half of 2009, but until June 30, 2009, there were none. The region reports, however, that positive findings from some transferred schools were used for advocacy . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): In 2007, the Bank hired for surveys through its supervision budget the Community Schools National Network . Simultaneously this organization was hired by the trust fund, which would seem to be a violation of trust fund covenants. Furthermore, the Bank had financed the existence of this network with Bank funds in 2004 and hired it it to conduct surveys. Trust fund documents prohibit this simultaneous financing due to potential conflicts of interest . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Unsatisfactory There were major deficiencies in the project’s relevance of design, achievement of objectives, and efficiency. Project design and implementation did not permit the testing of the hypotheses about the impact of formal transfers to community management or the development of a knowledge base on community-managed schools in Nepal Risk to Development Negligible to Low Significant There is no assurance of sustained Outcome : community involvement in transferred schools, particularly in poorer areas . Formal transfers have been scaled up without evidence of their effectiveness or equity implications. Bank Performance : Satisfactory Unsatisfactory Quality at entry was highly unsatisfactory, with unclear project objectives and incorrect definition of community schools. The Bank did not design a robust impact evaluation to establish cause-effect relationships between formal transfers of schools to community management features and student-related variables. Borrower Performance : Satisfactory Moderately The government was supportive Satisfactory through policies and legislation, but the Department of Education did not establish in a timely manner the administrative arrangements envisaged in the PAD. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: • Community management of schools offers many potential benefits, but it is important to understand what communities of different educational and income levels are likely to implement . The education of the poorest children should not depend heavily on the means available to the very poor communities . • Before scaling up pilot activities, it is important to evaluate those pilots using robust evaluation designs . In particular, Learning and Innovation Loans should strive to create best practice in evaluation in order to maximize learning. The failure to conduct adequate ex -ante stakeholder analysis can significantly reduce the scope for implementing reforms. Project implementers may thus fail to plan how to mitigate possible resistance to reforms 14. Assessment Recommended? Yes No Why? A PPAR was carried out in September 2008,of the CSSP and the Basic Primary Education Project II, as input for the Nepal CAE. 15. Comments on Quality of ICR: The ICR quality is unsatisfactory . Some of the reasons include the following : - The ICR does not systematically document the achievement of objectives, but rather the implementation of components and delivery of outputs (Section 3.2). In particular, no evidence is provided on the achievement of improved quality, efficiency, or accountability in the CSSP schools (although it is asserted that all improved ). Also, inadequate evidence is provided to assess the efficiency of the overall project (Section 3.3). Rather, the section re-counts all of the outputs and asserts that this was a lot of deliverables for $ 5 million. -The ICR does not consider alternative explanations for the reduction in out -of-school children (the only part of the objectives for which evidence is provided ). The large decline in out-of-school children occurs in 2006, the year that the insurgency ended (ICR, p. 10), yet there is no discussion of how this event might have affected outcomes . -Many conclusions in the ICR are not substantiated by evidence . For example, p. 29-30 state that the project 'triggered a process of transformation and change', that community management has "developed a culture of participatory decision-making", and that social audits improved governance in schools . - The ICR incorrectly represents the risk assessment shown in the PAD and incorrectly attributes increases in enrollment rates to the increased incidence of formally transferred schools (p. 9). The constraints listed on p . 31 are inconsistent with many assertions made elsewhere in the report . a.Quality of ICR Rating : Unsatisfactory