ICRR 11701 Report Number : ICRR11701 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 03/22/2004 PROJ ID : P035821 Appraisal Actual Project Name : District Primary Education Project Costs 534.4 483.3 Project II US$M ) (US$M) Country : India Loan/ Loan US$M ) 425.2 /Credit (US$M) 388.7 Sector (s): Board: ED - Primary Cofinancing 25.8 22.4 education (67%), Other US$M ) (US$M) social services (16%), Tertiary education (10%), Adult literacy/non-formal education (5%), General public administration sector (2%) L/C Number : C2876 Board Approval 96 FY ) (FY) Partners involved : Government of Closing Date 06/30/2003 06/30/2003 Netherlands Prepared by : Reviewed by : Group Manager : Group : H. Dean Nielsen Catherine Gwin Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives The Project Objectives were to: 1. Build national, state, district and subdistrict capacity to plan for and manage primary education; 2. Support state and district activities aimed at improving access to and the quality of primary education, including a) increasing enrollments, b) decreasing dropout, c) and increasing learning achievement, particularly for economically and socially disadvantaged groups, including girls, scheduled castes, scheduled tribes, and children with mild to moderate disabilities . b. Components The Project had the following four components (funding levels in US$ millions, excluding contingencies ): 1. Building State Institutional Capacity ($7.3); 2. Improving quality of, reducing dropout from, and expanding access to primary education ($483); 3. Developing a Distance Education Program ($4.3); 4. Continuing the National Management Structure Support for DPEP ($1.4). c. Comments on Project Cost, Financing and Dates Actual project expenditures were $ 483.3 million compared to the estimated costs of $ 534.4 million at appraisal, a rate of about 90 percent. The percent of IDA disbursed, based on the exchange rate (SDR to Indian Rupees) at appraisal, was about 92 percent, but correcting for devaluation of the Rupee, IDA disbursement was 100 percent. About 87 percent of the planned government counterpart funds were spent . Adjustments in the expenditure categories were made, including increases in IDA expenditures for civil works and staff salaries of 14 and 19 percent respectively, and decreases in all other categories, especially consultants (down by 85 percent). Overall project expenditures for teaching and learning materials were drastically reduced largely due to the greater than planned use of community procurement. The project closed on the original closing date . 3. Achievement of Relevant Objectives: On planning and management capacity : the Project established unprecedented patterns of annual work planning at all levels plus new progress monitoring systems; it also created a shift to quality improvement in the various government offices through the establishment and use of regular inservice teacher training; improved, field -tested learning materials; and school clusters and community groups (and the use of block grants ) for strengthening local commitment to and capacity for improvements in access and quality . There were differences across states and districts, such that some made impressive breakthroughs -- even to the point of spreading DPEP approaches beyond project borders, and others made little or no change . The specialized state and district agencies which were to provide technical support generally performed well, except for the District Institutes for Educational Training (DIETs) which often performed well below expectations . The monitoring and evaluation systems that were established seem to have sensitized stakeholders to the need for data collection, but the quality of the data and their use was weak . The performance of national level units was not well documented . On improving educational access and quality : a) Enrollments were reported to increase by 20 percent across DPEPII districts (for girls 25 percent and for scheduled casts and tribes 19 and 20 percent each, showing a possible closing of enrollment gaps for girls but not the other two target groups). These findings do not allow clear conclusions about Project effects, however, since they are not related to possible changes in the population -- changes in net enrollment rate would have been a more appropriate measure -- nor are there any control groups . Of those identified as having mild-moderate disabilities, about 77 percent were mainsteamed into schools, but this also a weak measure since no baseline condition is given . The ICR does not provide a full report on project -related increases in the number of students enrolled in community-based programs. b) Dropout : Sixteen percent of districts reduced dropout to the targeted 10 percent; in general across the states the level averaged around 30 percent (no data on baseline). There were serious questions as to whether the 10 percent target should have been adopted for all states and districts irrespective of their starting conditions . c) Achievement : For class 1, three years of program implementation was reported to boost scores in mathematics 40 percent over baseline and in language 23 percent; for class 3/4 the increases over baseline were 18 and 15 percent, respectively for mathematics and language; during the additional three years (2000-2003) increases of 13 and 12 percent were reported for class 1 mathemetics and language and 23 and 16 pecent for class 3/4 mathematics and language. It was not clear from the ICR whether these findings were cross sectional (i.e., for class 1, data from class 1 at baseline and then class 1 three years later) or longitutinal (data on class 1 at baseline and data on the same cohort three years later (when they are in class 4). Moreover, because of diffences between the baseline and final tests it was not possible to determine directly and conclusively whether the target of 25% improvement was reached. Finally, given no comparison group it is difficult to determine whether changes were related to the treatment or whether they were happening system wide . 4. Significant Outcomes/Impacts: The significant outcomes/impacts were similar to those for the DPEP I Project : a strengthening of government commitments at all levels to increasing school access and quality; the development of a strong, layered support system for community -based quality improvement in education; the development of a management information systems and the determination (if not yet the capacity) to use them for decision-making; demonstrated education system capacity to eliminate gender disparity in primary education access in most locations; breakthroughs in quality improvement through field -tested curricular/textbook revision and school -based inservice teacher training; an unprecedented level of attention given to children with mild to moderate learning disabilities . 5. Significant Shortcomings (including non-compliance with safeguard policies): Significant shortcomings were: weakness in the quality of data generated by the evaluation /information systems set up under the project, such that the impact of the Project on goal attainment, especially on improved access and quality, was poorly documented; weaknesses at entry: the targets related to project outcomes were not specified in the objectives (as in the case of DPEP 1) and the baseline data tables in the SAR Annex 5 were mostly empty. Growth of enrollment and its measurement (the use of enrollment rates) were not well conceptualized; low priority given to alternative school modalities (e.g., non-formal primary classes, the expansion of early childhood education opportunities, targeted interventions for children with mild -moderate disabilities, and the use of distance education for inservice teacher training and other training (component 3)), and the relative lack of data on these component/subcomponent's activities and outcomes; the lack of attention in Project supervision (e.g., in Joint Review Missions) to sector-wide policy issues and financial monitoring (as envisioned in SAR); relatively weak performance of the District Institutes of Education and Training in many states, despite the fact that this was identified as a problem in DPEP 1. 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Satisfactory [The ICR's 4-point scale does not allow for "moderately sat." rating]. The Project set up a multilayered structure for monitoring and evaluation but failed to generate clear findings on the achievement of targeted results, especially concerning increased enrollments, reduced access disparities, school drop-out, and mainstreaming of learning disabled. The existing data tend to show that drop out targets were not reached in most districts. Lacking any control group information, it was difficult to tell whether changes were due to project treatments. Institutional Dev .: Substantial Substantial Sustainability : Likely Likely Bank Performance : Satisfactory Satisfactory Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. NOTE: 7. Lessons of Broad Applicability: This project made important contributions in showing : the power of strong government ownership of the various intervention strategies at all levels coupled with strong support from resident Bank professionals; the desirability of keeping the project design simple and focused : the core of this project -- district level changes in educational access and quality, measured by simple indicators -- made sense and allowed for focusing of efforts; the peripheral parts of the project (nonformal education for those too far from schools, mainstreaming the disabled, strengthening early childhood education programs, and distance education for inservice training ) were barely reported on and may have been better treated in separate projects . the desirability of building local level institutions /channels to deliver services (block and cluster resource centers ) supported professionally by existing entities at the district, state, and national levels; the positive impact of good analytical work during preparation coupled with the government's determination to foster change based on experience and field -based lessons; the contraversial nature of building international technical assistance into project loans in countries like India, which have a strong domestic skills base . In the end, the GOI decided to forego the use of DPEP international technical assistance. It would be useful to know why this decision was made and whether it had any impact on the quality of the Project and its implementation . If such technical assistance would have been useful but the decision was made because of its high expense relative to other inputs, there may be a strong case for finding ways of financing international TA outside of project loans . 8. Assessment Recommended? Yes No Why? The scope of the seven DPEP projects in India (this Project plus the six others ) is huge, allegedly effecting educational opportunities and outcomes among hundreds of thousands of children; nevertheless, real Project outcomes and impacts are obscured by poor quality of outcome data . It would be useful to get insight into the full impact of the Project and to understand more fully how complex, multi -layered networks of implementation and support can be energized . 9. Comments on Quality of ICR: The ICR creates confusion at the beginning since its versions of project objectives and components are quite different from those in the Staff Appraisal Report (SAR)-- for example, the ICR elaborates on SAR objectives, sometimes uses different terminology, adds minor elements and omits others . Concerning the reporting of project accomplishments, the ICR is often weak in its use of facts and evidence, often presenting what was expected to be accomplished instead of what was accomplished . Finally, many of the paragraphs are poorly written and difficult to understand.