ICRR 14264 Report Number : ICRR14264 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/28/2014 Country : Indonesia Project ID : P115724 Appraisal Actual Project Name : Basic Education US$M ): Project Costs (US$M): 40.83 17.35 Capacity-recipient Executed L/C Number : Loan/ US$M): Loan /Credit (US$M): 20.53 17.35 Sector Board : Education Cofinancing (US$M): US$M ): Cofinanciers : Board Approval Date : 06/06/2007 Closing Date : 04/30/2012 01/27/2013 Sector (s): General education sector (100%) Theme (s): Education for all (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Peter D. Bachrach Judyth L. Twigg Lourdes N. Pagaran IEGPS2 2. Project Objectives and Components: a. Objectives: In lieu of a Project Appraisal Document, a Grant Appraisal Document (GAD) was prepared and established the overall objective of the project : “to contribute to the Government of Indonesia ’s achievement of Millennium Development Goals (MDG) and Education for All (EFA) goals through good governance in education � (p. 6). The trust fund agreements with both the Netherlands and the European Commission (EC) state the objective slightly differently: “to support the GoI’s achievement of MDG and EFA goals through better governance in education.� The Implementation Status Reports (ISR) further develop the objective : “To support the Government of Indonesia’s (GoI) achievement of MDG and EFA goals through good governance in education, improving the delivery of decentralized basic education services by local governments and schools in selected program areas and by extension in other locations .� The Grant Agreement (page 5) between the Bank and GoI states the objective to be “to contribute to the achievement by the recipient of Millennium Development Goals (MDG) and Education for All (EFA) goals through good governance in Indonesia .� This review uses the objective in the Grant Agreement . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: Component 1: Policy Analysis and Dialogue (Appraisal: US$1.1 million; Actual: n/a). This component was to support: (i) a forum for key stakeholders to discuss education sector issues, including strategic and policy recommendations for reforms and to review sector performance; (ii) dialogue to address coordination among government institutions and between GoI and its partners; and (iii) preparation of a sector-wide approach. The component was also to support research to assess the education sector and analysis to support movement to a sector-wide approach. Component 2: Good Governance, Financial Management, Budgeting and Accountability (Appraisal: US$10.7 million; Actual: n/a): Financing of sub-components to support: (i) financial management and good governance activities at the district level; (ii) the development of results-based planning, performance-based budgeting (and a Medium Term Expenditure Framework), and simplified reporting; and (iii) development of capacity for school management. Component 3. Information Management (Appraisal: US$9.6 million; Actual: n/a): Financing of sub-components to: (i) implement management information systems by developing the capacity to manage and analyze data and to make informed decisions based on results; and (ii) facilitate knowledge-sharing and learning about good practices. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost: Actual component costs are not available because the project implementation unit did not keep disbursement records by component, only by category . The significant difference between appraisal and actual project costs results mostly from the reduction of the trust fund amount by €8.7 million at the request of the Dutch government . Financing: A sector-wide program was initially envisioned, but country systems at the time were not deemed sufficiently strong. As a result, the Netherlands, the European Commission (EC), and the World Bank established the Basic Education Capacity Trust Fund (BEC-TF) to support the education reform agenda . Additional grant funds were also made available by the Dutch Education Support Program (DESP) to fund discrete activities. BEC-TF’s five-year financial commitment from the Dutch and the EC amounted to €39 million, which was divided into two parts: €25.60 million was Government-executed, and €13.40 million was administered by the Bank. A level two restructuring was carried out (October 2011) to: (i) address the cancellation of certain pilot activities initially covered by the project (and subsequently financed by other sources ); and (ii) reallocate grant proceeds due to exchange rate fluctuations . In October 2012, the GoI requested the cancellation and transfer of €1.1 million from the unallocated disbursement category back to the parent trust following lower than anticipated disbursements. As of May 2013, approximately US$3.2 million (resulting in part from Euro exchange rate gains over the life of the project) was undisbursed. Borrower Contribution: There was no planned Borrower contribution . However, during the implementation of activities, the Central Project Implementation Unit (CPIU) and districts provided matching funds sourced from national and regional budgets to aid the implementation of operational activities at the district level . The amount of matching funds provided by the CPIU amounted to Rp 24,461,404,000 (US$1.957 million). These funds were used for coordination, facilitation, study visits, mentoring, and monitoring and evaluation . The amount of matching funds provided by each district varied according to the budget capacity of each district and ranged between 2 and 25 percent of the total Local Government Basic Education Capacity (L-BEC) budget. The total value amounted to Rp 15,717,413,443 (US$1.257 million) over the three years of the project . Dates: The project closing date was extended from April 30, 2012 to December 31, 2012 to: (i) make up for initial implementation delays; and (ii) enable the participating districts to complete planned 2012 capacity development plans (CDPs) contributing to progress on the project ’s Key Performance Indicators (KPIs). 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial, The Bank's Country Partnership Strategy (CPS, FY13-15), current at project closure, notes that the Government, after the successful launch of the Bantuan Operasional Sekolah (School Operational Assistance ) or BOS program, no longer intended to use Bank financing for basic education; rather, it wanted Bank support to improve the quality of spending and to provide grants to address pressing needs in the most deprived areas . The project under review was consistent with these principles : it was financed by Trust Funds and aimed to improve the quality of spending on education through decentralized planning and budgeting to meet educational needs. Selection criteria included poverty and other local government characteristics (governance, public financial management, and education performance indicators ). Among the engagement areas cited in the CPS, the project contributed directly to the educational objectives (particularly to improving the quality of instruction and the performance of teachers ) and less directly to other areas: local government and cross-cutting engagements (gender and governance areas ). The Government continues to support the objectives of education sector reform and the Five -Year Strategic Plan (RENSTRA) as the means for ensuring higher quality of management of primary and junior secondary education. The EC and the Dutch Government promoted a sector -wide approach (SISWA) to implement the plan, but the approach was eliminated due, as the project team noted in its meeting with IEG, to a lack of consensus on the framework for such an approach . b. Relevance of Design: Modest . The Grant Appraisal document (p. 41) presented the logical relationship between the project outputs and their linkage to the educational determinants for results and the expected outcomes . A careful reading of this detailed description indicates that the project ’s measurable outcomes were directly linked to educational access and quality and only indirectly linked to good governance and accountability . In addition, the initial design’s focus on governance, planning and budgeting, financial management, and information and accountability was subsequently often modified; the Grantee ’s ICR describes a dozen changes between 2008 and 2012. These changes involved virtually every aspect of the original design : precision of the objectives, criteria for education capacity, procedures for grant agreements and flow of funds, modification of the implementation manual, limited flexibility in the three -year capacity development plans, etc . According to the Grantee’s ICR, the design contributed to confusion in : (i) the relationships between the GoI, the World Bank as Trustee, and the donors contributing to the Trust Fund; (ii) the roles of the various ministries for financial and technical management; and (iii) responsibilities at the district level . In a meeting between IEG and the project task team, the team concurred that splitting execution of the trust fund (one-third Bank and two-thirds Indonesia) did not work well, either for disbursing funds or for coordinating responsibilities . 4. Achievement of Objectives (Efficacy): Consistent with the GOI’s overall goal of accelerating Quality Basic Education for All through a medium -term strategy (as spelled out in the RENSTRA and anchored on the three pillars of access, quality and governance ), the PDO was “to contribute to the Government of Indonesia ’s achievement of MDG and EFA goals through good governance in education.� The Grant Appraisal Document included a Logical Framework but not a Results Framework with baseline and target values. The ICR (pp. iii-vi) includes baseline, target, and actually achieved values for both the PDO and Intermediate Outcome indicators, although the sources of information are not indicated in the Data Sheet . PDO indicator results were provided by the Directorate General of Basic Education, are national in scope, and differ slightly from the information presented in the Data Sheet, which does not provide information on the data sources (although there is a general discussion of sources in the M&E section (p. 9)). The Intermediate outcome indicators were derived from the Survey of the Quality of Local Education Governance (published by the Bank in 2013) and concern only the project ’s education districts (50 of 483 nationally). Achievement of the project's broad objectives -- to contribute to the achievement by the recipient of Millennium Development Goals (MDG) and Education for All (EFA) goals -- is rated according to the three outcomes covered across the education -related MDGs and EFA goals: universal primary schooling, measurable learning outcomes, and elimination of gender and other disparities . In a meeting with IEG, the project team clarified that the MDG goals had been quantified but the EFA goals constituted broad goals that were incorporated into policy and strategy statements . The project contributed to MDG goals 2 and 3 and to EFA goals 2, 3, and 6, specifically: (i) access (through universal primary schooling ); (ii) quality (through measurable learning outcomes ); and (iii) equity (through the elimination of gender and other disparities in primary schooling). The project expected to contribute to these goals by supporting a range of outputs aiming to increase good governance (in primary education). Outputs relevant to all three objectives Policy analysis and dialogue : • Rapid reviews conducted by the Bank at the request of GoI (e.g., education financing, school management, regulatory framework for early childhood development, etc .) and intended for use by the government to inform its own sector policy dialogue. • Analytical studies, including an Education Sector Assessment, a Youth Employment Strategy, and an information and communications technology (ICT) and distance learning regulatory framework . • Annual review and updating of the school operational assistance (BOS) operations manual necessary for the allocation and utilization of BOS funds under the provincial block grant mechanism . • Local and international workshops for professionals and policy -makers on selected issues . Good Governance, Financial Management, Budgeting and Accountability : • Formulation of three-year capacity development plans using grant funds to address local (rather than centrally determined) needs. • Implementation of a capacity development program comprising : (i) training modules and a trainer’s guide in the project’s strategic areas: Transparency and Accountability; Education Service Provision Standards, Management Control Systems, Education Management Information Systems (EMIS), and Efficient Resource Use; and (ii) training of stakeholders and local authorities in all 50 districts. • Development of costing tools to guide districts in their planning and budgeting processes, including public expenditure tools, Excel-based costing applications, and allocation formulas . • Strengthening of the budget process with budgets related to measurable outcomes, disclosed locally, and reported on periodically. • Preparation of a detailed manual for implementing the grants program, with support to the districts for the preparation, submission, and evaluation of the grant proposals . • Comprehensive study and analysis of best practices for school -based management. • Training on assets management and financial audits . Information Management • Development of a project M&E system and training for District Education Officers and School Supervisors . • Implementation of a complaint handling system based on community participation and involving training and sensitization of local authorities and development of online and text message -based feedback systems . • Establishment of learning and knowledge -sharing systems, including: (i) the Tool for Reporting and Information Management by Schools (TRIMS) for evidence-based policy development; (ii) the WAPIK program, an information network providing information and materials to mainstream good practices . • Formulation and implementation of a communications strategy to inform stakeholders and increase the project’s visibility. Outcomes Universal primary schooling (access ): Substantial, based on increases in primary gross and net enrollment rates . Increased primary gross enrollment rates from 99.00 percent in 2007 to 102.96 percent in 2012, falling somewhat short of the original target of 103.15 percent. Increased primary net enrollment rates nationally from 91.13 percent in 2007 to 92.52 percent in 2012, exceeding the original target value of 92.34 percent. (MDG 2.1) According to the ICR, there were increased junior secondary level net enrollment rates (between 2007 and 2012) among both males (rising from 64.37 percent to 68.09 percent and virtually matching the target of 68.55 percent) and females (rising from 68.68 percent to 71.95 percent and exceeding the target of 68.75 percent). The project team subsequently provided additional data differentiating project from non -project districts: in non-project districts, junior secondary gross enrollment rates remained stable at 89% from 2010 to 2012, while over the same time period gross enrollment rates in project districts increased from 86% to 93%. Similarly, junior secondary net enrollment rates remained stable at 70% for non-project districts from 2010 to 2012, while increased for project districts from 68% to 71% over that same time period. Measurable learning outcomes (quality ): Modest, based on overall slight increases but failure to achieve targets . In the absence of explicit outcome indicators, this Review uses the data in the ICR data sheet on completion rates and transition rates for primary schooling . Increased primary completion rates (between 2007 and 2012) from 96.86 percent to 97.06 percent but short of the end-of-project target of 97.70 percent. (MDG 2.2) Increased transition rates from primary to junior secondary school from 75.58 percent to 77.26 percent between 2007 and 2012, short of the target of 82.20 percent. Increased junior secondary completion rates from 98.17 percent to 98.27 percent between 2007 and 2012, but short of the target of 98.61 percent. Elimination of gender and other disparities (equity ): Substantial, based on overall increases in attendance ratios among primary female and poor students . In the absence of explicit outcome indicators, this Review uses the data in the ICR data sheet on gender and poverty for primary schooling . Increased primary gross enrollment rates for females from 98.43 percent to 102.86 percent between 2007 and 2012, exceeding the target of 102.53 percent. The proportion of primary female students to primary male students increased from 99.40 percent to 99.81 percent. Increased primary net enrollment rates for female students from 90.85 percent to 92.56 percent between 2007 and 2012, exceeding the target of 92.11 percent. The proportion of primary female students to primary male students increased from 98.89 percent to 100.1 percent. (MDG 3.1a) Increased primary net enrollment rates for the poorest wealth quintile from 90.93 percent in 2007 to 92.52 percent in 2012. Decreased junior secondary school female /male net enrollment ratio from 106.70 percent in 2007 to 105.67 percent in 2012. (MDG 3.1b) Increased junior secondary gains within the poorest income quintile from 53.03 percent in 2007 to 63.54 percent in 2012. Assessing these outcomes and the project ’s contribution to them is complicated by the fact that : (i) the outputs are only indirectly linked to the outcomes (as shown in the GAD’s logical relationship diagram and in the lists of outputs and outcomes above ); and (ii) the attribution of the national outcomes to the project is problematic because the project covered roughly 10% of the education districts nationally . The ICR (p. 6) notes this problem, stating that the PDO indicators were included at the specific request of the EC and the Dutch and were based on regularly collected data. On the other hand, the project ’s intermediate outcome results are almost certainly due to the outputs financed by the project . As clarified by the project team in a meeting with IEG, the average annual grant amount of US$ 80,000 per district was established in agreement with the districts and was intended to provide an incentive to improve gove rnance. The project team also clarified that almost half of project funds were spent on capacity-building and policy development activities at the national level, and therefore national-level indicators are reasonably appropriate to assess outcomes resulting from the project's interventions. 5. Efficiency: Efficiency : Modest The ICR (Annex 3) offers eleven arguments to establish its conclusion on the efficiency of the project, but more than half of these address efficacy more than efficiency . Two could be excluded: (i) internal efficiency gains, because these are national data; and (ii) unit training costs, since these are only a small part of capacity building. Given the availability of grant expenditure data and the results of the Local Governance and Education Performance survey (especially, the correlation coefficients between selected variables and the district index scores), a more rigorous analysis of efficiency would seem to have been possible . The ICR (Annex 3) argues that: (i) there have been important gains in the processes determining the use of public resources; and (ii) leakage has been significantly reduced . In both instances, the ICR notes that improvements in the project districts were at least in part the result of GoI ’s overall efforts to improve planning (through the introduction of the medium -term expenditure frameworks) and strengthen internal and external audit systems. The ICR's discussion of efficiency does not address project management efficiency, and particularly the effects of the 18-month delay in L-BEC grant approvals, implementation constraints associated with the management and flow of funds, and the impact of the reduction of funds by the Dutch. The project team provided additional information that, during the 18-month delay in grant approvals, the project team efficiently carried out a wide range of activities designed to prepare district -level staff for eventual implementation of grant funds; the delay in approving the grants therefore did not result in a significant overall slowdown in release of project funds . Even with this new information, however, there is still insufficient information on the cost -effectiveness of allocation of Bank resources. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of project objectives is rated Substantial, based on the conformity of the project with current policies, strategies, and initiatives . The relevance of project design is rated Modest, based on the difficulties introduced by the split responsibilities (and grant amounts) for implementation. Achievement of the MDG and EFA goals to increase access and equity were substantially achieved, but there is little evidence provided of substantial progress on measurable learning outcomes, and the project districts did not meet targets in this area . Efficiency is rated Modest, due to insufficient information on cost -effectiveness of allocation of project resources . Taken together, these ratings are indicative of moderate shortcomings in the project's preparation and implementation, resulting in an Outcome rating of Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Programmatically, GOI remains committed to achieving the MDGs and the EFA goals . The RENSTRA provides a coherent policy framework for stakeholders to set priorities, define gaps, and pilot and scale up new modalities and for strengthening harmonization and alignment . The BOS was initiated in 2005 and constitutes a means for alleviating fees for all students and eliminating fees for poor students . The government intends to include lessons learned from the project on transparency, financial management, and budget planning into the overall BOS operations manual for use nationwide . Managerially, education district authorities have adopted planning and budgeting measures to respond to both local needs and central/provincial requirements. The project contributed to the Ministry of Finance's development of an efficient funding flow mechanism that transmits funds from the center to the provinces and then the district and schools . Financial management has been improved, and audits at the district level report reduced levels of financial leakage during the 2008-2012 period. Given the complexity and size of the Indonesian education system, there are still challenges with financial reporting and internal and external control systems. Information collection tools have provided local schools and communities with increased capacity to monitor and evaluate the quality and efficiency of school development programs . Financially, according to the project team in its meeting with IEG, the project ’s Advisory and Analytical Activity (AAA) work contributed to regulatory changes in education and to an overall doubling in spending on education, including support for primary and junior secondary education . In addition, 40 of the 50 project districts have developed post-project sustainability plans and have mobilized funds to continue : (i) the Education Management Information System, including the Tool for Reporting and Information Management by Schools (TRIMS); (ii) financial management, including performance -based budgeting and planning, asset management, efficient resource use, and procurement policies, procedures and controls; and (iii) School-Based Management Training, including school committee development and community involvement programs . Funds have also been committed to : (i) reform of district education policies and regulations (including strengthening the Province/District Operational Assistance Program (BOSDA)); (ii) strengthening of Education Service Provision Standards, including teacher upgrading and competency -based curriculum development; and (iii) initiating incentive schemes to encourage the deployment of qualified teachers to underserved provinces and districts. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The ICR (p. 6) notes that: (i) the project fit squarely within the goals of the FY 04-07 CAS and the RENSTRA five-year plan; (ii) preparation took advantage of lessons from earlier Bank -supported capacity development programs and included relevant study tours; (iii) the Bank team worked effectively with relevant ministries to prepare the project; and (iv) preparation relied on a bottom-up approach through the development of the Local Governance Capacity Assessment and the Capacity Development Plans . Less clearly described in the Grant Appraisal Document are the operational implications of the management of the trust funds, where about two-thirds were executed by GoI and one -third by the Bank. While no QER was conducted, the ICR (p. 7) notes that: (i) project preparation followed the Bank ’s comprehensive quality assurance and review process; and (ii) the government met all project readiness conditions by the effectiveness deadline . There were, however, shortcomings related to quality at entry . The financial arrangements, which split responsibilities for executing the trust fund, were never fully resolved and contributed to confusion and misunderstanding. Also, the absence of a final agreement during preparation to adopt a sector -wide approach significantly affected the PDO; the PDO was established at the program level, but did not provide project-level objectives/indicators that could be attributed to the project . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The ICR (p. 20) notes the positive effects of : (i) the proximity and the continuity of leadership by the Jakarta-based Bank team, from preparation through implementation; (ii) the regularity of formal supervision missions; and (iii) the presence of Bank financial management and procurement staff in all six supervision missions. Bank supervision and government authorities proactively countered the negative effects of delays in introducing the on-granting mechanism by adjusting the sequencing and rolling out of activities . In addition, Bank supervision participated positively in the frequent changes occasioned during implementation by modifications in the design and volume of activities carried out . Supervision was not effective in aligning the program PDO and project objectives with the changing situation of the project. There were two occasions where the Bank could have addressed the problem of alignment of outcomes: (i) at the time of the amendment of the administrative agreement between the Netherlands, EC, and the Bank (in February 2008); and (ii) at the time when the Netherlands and EC withdrew, thereby reducing significantly the resources of the project (in October-November 2011). This was not done (though by 2011 it may have been too late), with the result that the disconnect between program PDOs and project objectives persisted throughout the project . The project team later stated, however, that this disconnect was not a significant issue, as the project funded a significant amount of policy development at the national level that would plausibly have been expected to contribute to national -level outcomes; it is still the case, however, that there were no indicators for monitoring and reporting on outcomes specific to the district -level activities that the project funded through Components 2 and 3. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: At the central level, the government was inconsistent in its commitment to the SWAp, first expressing interest in it (ICR, p. 2) and then declining to participate (but after Board approval of the project ) (ICR, p. vi). In addition, delays in approving the regulation for the new on -granting mechanism delayed funding for the districts by eighteen months or almost half of the project period . Even then, at the district level, the transfers of grant funds, the documentation required to access the funds, and the disbursement procedures were all constraints in implementing the planned activities . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Grant Appraisal document described the units to be established at central, provincial, and district levels to implement the project. Keys to project coordination were the Central and District level PIUs (CPIU and DPIU). Based on an analysis of their functions and tasks, the ICR (p. 21) rates performance as moderately satisfactory. The Grantee’s ICR is somewhat more critical, noting that staffing changes, modifications in the nature and sequencing of the units ’ activities, and insufficient funding hampered the overall performance of the CPIU and specifically its technical support to the districts . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project’s results framework was closely aligned with the donors' and government ’s desire to monitor progress towards achieving the MDG and EFA goals . Data sources to monitor achievement of the PDO -level indicators relied on established data sources, including the National Socioeconomic Survey (SUSENAS) and the “Education Statistics in Brief� published by Ministry of Education and Culture (MoEC). However, these instruments were not able to provide data on the MDG and EFA outcomes specifically for the fifty districts targeted by the project. The intermediate indicators were aligned with the government ’s RENSTRA, and measurement relied on the Local Governance Capacity Assessment (LGCA) and the Indonesia Local Education Governance Index (ILGI) to provide a coherent national capacity development assessment of performance in education governance and service provision in the decentralized environment . The LGCA survey provided the Bank and government with a data set to systematically compare changes in local education governance within the 50 BEC-TF districts between 2009 and 2012. In addition to helping districts assess their capacity development needs, the LGCA survey provided a supplementary and robust source of data to check information gathered by the districts and reported to the CPIU and the Bank . In addition, the project funded the development of several M&E tools, including : (i) the TRIMS to provide a systematic approach to gathering and reporting data at the school level; (ii) a complaint handling mechanism at the school and district level; and (iii) the “SupportMySchool (SuMS)� program to deploy an online system for the public and schools to report critical needs and complaints . Finally, the project included other forms of monitoring and evaluation, including education assessments, public expenditure analysis, school management, and studies and policy briefs on various subjects . b. M&E Implementation: The surveys intended to measure the PDO were conducted as planned . M&E, as noted in the ICR (p. 10), was developed in 2009 but not finally adopted until 2011 when it was disseminated to District Education Office staff in a three-day training session. The TRIMS was introduced to all schools in 2011 as part of the EMIS reforms, and 4,993 schools are reporting data; SuMS was rolled out to 497 districts and 33 provinces by 2012. c. M&E Utilization: The ICR (p. 36) concludes that M&E “utilization is uneven and data results used to inform policy and budgetary decisions is also uneven .� However, during the project, results framework data were used for discussions as part of the Thematic Education Dialogue (TED) and for Bank reporting in the ISRs after supervision missions . Subsequently, the key M&E tools have been nationally disseminated, are available for use on dedicated web sites, and have been followed up by both MoEC and a local university . Since December 2012, there have been over 4,000 users of the SuMS, and MoEC is slowly adapting the system to highlight successes, ongoing challenges, and constraints for districts . In addition, the findings of several policy studies supported by the project were used by the government to support education reforms . The CPIU monitoring and evaluation of the L-BEC grant program implementation at the district level occurred up to three times each year and included representatives from MoEC, the Ministry of Religious Affairs, and the Ministry of Finance . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: Compliance with environmental safeguards was rated satisfactory . BEC-TF was rated a category “C� project based on the fact that no safeguards policies were triggered . b. Fiduciary Compliance: Financial management: Disbursement was negatively affected by an eighteen -month delay in approving on-granting regulations. In addition, Financial Management specialists found some early weaknesses with compliance and more serious problems of questionable expenditures for a series of workshops; both of these issues were corrected by GoI . The GoI Financial and Development Supervisory Board conducted audits at the national and district levels, which were unqualified and presented on time . Procurement. Procurement was carried out by the CPIU and the districts (based on guidelines laid out in the BEC-TF Operations and L-BEC Manuals) and was limited to: consultants; goods (mostly computers and data processing equipment); workshops, meetings, training, and study tours; and incremental operating costs . A 2011 Bank Trust Fund Portfolio Review found that procurement was carried out in compliance with applicable guidelines and procedures, though several procurements had to be cancelled as a result of the delay in the on-granting mechanism. c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons adapted from the ICR (pp. 21-22): The use of large scale longitudinal surveys such as the LGCA provides a good capacity development needs assessment tool to a project . Grant planning, submission and approval, disbursement, and expenditure verification are time -consuming but contribute to a more effective district response to its needs, improved financial management practices, and better transparency and accountability . When decentralized planning and budgeting are envisioned as a means for improving services, they must be accompanied by a clear funds flow mechanism to ensure that there are no delays in implementing the plan. Lessons from IEG: Where a project’s essential objective is to develop district capacity, the ICR must demonstrate not only what capacity has been developed but also how this occurred and varied across districts . 14. Assessment Recommended? Yes No Why? Capacity development is not often operationalized and rarely measured before and after an intervention . In this case, both conditions were met for a population of fifty districts . If analysis is not ongoing, it might be interesting to examine in more detail the methodology and its applicability elsewhere . 15. Comments on Quality of ICR: The ICR Quality is rated Satisfactory . Annex 2 (Outputs by Component) is particularly well done. In addition, the Grantee’s ICR is very useful for understanding the project . The Bank ICR is well written (though somewhat repetitive) and candid about the achievements and challenges of the project. There are, however, several gaps in the analysis : • The Data Sheet would have benefited from citing the sources of information . • The ICR does not include students among the beneficiaries and has little to say about them later (p. 4) in the ICR. • There is not much discussion about how project preparation could have missed the government ’s lack of interest in a SWAP and of an appropriate regulatory framework for the on -granting mechanism. • There is no discussion of risks and risk mitigation measures . It is unfortunate that the results of the Survey of the Quality of Local Education Governance were not published until October 2013, since more analysis of the changes in and variation among district performance could have enabled attribution of observed outcomes to the project's interventions . A few tables and somewhat shorter paragraphs would have helped to navigate the text . a.Quality of ICR Rating : Satisfactory