Document of The World Bank FOR OFFICIAL USE ONLY Report No: 21496 IMPLEMENTATION COMPLETION REPORT (IDA-23540) ONNA CREDIT IN THE AMOUNT OF US$115 MILLION EQUIVALENT TO THE ISLAMIC REPUBLIC OF PAKISTAN FOR THE PUNJAB MIDDLE SCHOOLING PROJECT JANUARY 23, 2001 EDUCATION SECTOR UNIT SOUTH ASIA REGION This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective February, 1992) Currency Unit = pakistan Rupee (Rs.) Rs. 1.00 = US$ 0.0406 USS 1.00 = Rs. 24.63 FISCAL YEAR July 1 - June 30 ABBREVIATIONS AND ACRONYMS ADP Annual Development Plan CAS Country Assistance Strategy CEC Civil Engineering Cell CRDC Curriculum Research and Development Center C&W Construction and Works DEOs District Education Officers DDEOs Deputy District Education Officers DflD Department for International Development DMG District Management Group DoE Department of Education DPI-E Department of Public Instruction-Elementary DPI(EE) Department of Public Instruction (Elementary Education) DSD Directorate of Staff Development ECNEC Executive Committee of the National Economic Council EDU Education Development Unit EMIS Education Management Information System GCET Government College for Elementary Teachers ICR Implementation Completion Report NGOs Non-Govemmental Organizations PC-I Planning Commission: Initial Project Document PC-IV Planning Commission: Project Completion Document PIU Project Implementation Unit PEMIS Punjab Education Management Infomnation System PMSP Punjab Middle Schooling Project PMU Project Management Unit SAPP Social Action Program Project SAR Staff Appraisal Report SDR Special Drawing Rights SMCs School Management Committees SNE Schedule of New Expenditures TA Technical Assistance TDC Test Development Center USAID United States Agency for Intemational Development Vice President: Mieko Nishimizu Country Manager/Director: John W. Wall Sector Manager/Director: Emmanuel Y. Jimenez Task Team Leader/Task Manager: Fredrick L. Golladay FOR OFFICLAL USE ONLY CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 1 4. Achievement of Objective and Outputs 3 5. Major Factors Affecting Implementation and Outcome 7 6. Sustainability 8 7. Bank and Borrower Performance 9 8. Lessons Learned 12 9. Partner Comments 14 10. Additional Information 16 Annex 1. Key Performance Indicators/Log Frame Matrix 17 Annex 2. Project Costs and Financing 18 Annex 3. Economic Costs and Benefits 20 Annex 4. Bank Inputs 21 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 24 Annex 6. Ratings of Bank and Borrower Performance 25 Annex 7. List of Supporting Documents 26 Annex 8. Government's Comments 28 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not be otherwise disclosed without World Bank authorization. Project ID: P010394 Project Name: MIDDLE SCHOOLING Team Leader: Fredrick Golladay TL Unit: SASED ICR Type: Core ICR Report Date: January 23, 2001 1. Project Data Name: MIDDLE SCHOOLING LIC/TFNumber: IDA-23540 Country/Department: PAKISTAN Region: South Asia Regional Office Sector/subsector: ES - Secondary Education KEY DATES Original Revised/Actual PCD. 01/30/90 Effective: 09/04/92 03/02/93 Appraisal: 01/01/92 MTR: 05/20/97 Approval: 04/14/92 Closing: 06/30/2000 06/30/2000 Borrower/ImplementingAgency: GOVT OF PAKISTAN/PUNJAB EDUC. DEPT. Other Partners: Department for Intemational Development (DFID), UK STAFF Current At Appraisal Vice President: Mieko Nishimizu Joseph D. Wood Country Manager: John W. Wall Michael Wiehen Sector Manager: Emmanuel Y. Jimenez Roberto Cuca Team Leader at ICR: Tahseen Sayed Khan Daniel Viens ICR Primary Author: Fredrick L. Golladay 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: M Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The original objectives of the Project were to: (a) increase access and participation in middle schooling in rural areas and urban slums, especially among girls; (b) improve education quality and raise student leaming and achievement; and (c) strengthen related policy-making, planning, management, monitoring and evaluation capacities. 3.2 Revised Objective: The Project was restructured in September 1997 as part of a Pakistan-wide effort to better focus the social sector portfolio in view of overall flagging performance and to ensure better alignment of provincial social sector projects with the Goverrnent's National Social Action Program. Targets for increasing capacity to provide middle schooling were reduced in line with reassessed demand for middle schooling and departrnental capacity to implement civil works. Initiatives aimed at raising the quality of education were expanded to cover innovative activities and to benefit grades K-5 as well as 6-8 (middle schools). Additional assistance was made available to the Department of Education for strengthening policy-making, planning and management, and community-based management. 3.3 Original Components: Increase access and participation. Access to schools was to be increased for children (particularly girls) living in rural areas and urban slums by constructing approximately 14,100 additional classrooms, and providing facilities such as latrines, drinking water supplies and security walls to about 1400 schools. About 70% of beneficiaries were expected to be girls. Girls'participation in middle schooling was to be further encouraged by providing 12,000 scholarships a year to qualified, needy students and by increasing the supply of female teachers available in rural areas through a program to grant 1,000 scholarships a year to rural girls pursuing secondary education and teacher training. In addition, the role of local government in the finance, management and delivery of elementary education was to be expanded on a pilot basis through technical assistance. The involvement of communities and the private sector in the delivery of elementary education was to be promoted by strengthening the Punjab Education Foundation. Improve education quality. The effectiveness of teachers and headteachers was to be increased through in-service training in pedagogy; mathematics, science, and English teaching; school administration; and school finance. The quality of science education was to be raised through the development of low-cost demonstration materials and supply of materials, equipment and training. Capacity for the assessment and improvement of learning was to be strengthened through the development of expertise in testing and assessment, and the institutionalization of practices designed to monitor the achievement of progress in meeting goals for curricula. The effectiveness of teaching and learning materials was to be increased through technical assistance in the preparation and publication of new textbooks, teachers' guides and supplementary reading materials, all of which were to be grade and context sensitive. Finally, incentives for improving staffperformance were to be offered through a program of awards and recognition of outstanding teachers and headteachers. Strengthen capacity for policy-making, planning, management, monitoring and evaluation. Capacity for policy formulation and planning was to be developed at the provincial level through creation of, and technical assistance to, an Education Development Unit (EDU). Capacity for collecting and interpreting management information was to be increased through the creation of a provincial education statistics unit, preparation of a geographic information system and strengthening of capacity for monitoring and evaluation. Attention to elementary education was to be increased through administrative reforms separating responsibility for the management and administration of elementary and secondary education at provincial, divisional and district levels. 3.4 Revised Components: Restructuring of the Project was undertaken to (a) better integrate the initiatives being pursued by provincial projects into the national policy framework created by the Social Action Program; and (b) reduce Project targets in view of the slow pace of project implementation during the first four years of the Project and the emerging evidence that many fewer middle schools could be constructed within the guidelines for siting of schools adopted under the Project. - 2 - Increase access andparticipation. The number of middle schools to be created, expanded or upgraded was reduced from 3600 to 2400 (still an ambitious target in light of the time remaining). Improve the quality of education. The components of the Project intended to increase the quality of education were expanded in both scale and scope. In particular, the coverage of the program to raise the quality of education was expanded to include primary as well as middle schools and explicit provision was made for supplying supplementary reading materials to primary schools. The focus of the Project was shifted from construction to raising the quality of education. Strengthen policy-making, planning, management, monitoring and evaluation. The restructured program offered additional support to the Department of Education (DoE) for the establishment of the Education Development Unit (EDU)--a planning cell. The Project also provided assistance for the training and support of School Management Committees (SMCs) composed of stakeholders drawn from the community. Finally, the restructured Project allocated additional funds to the support of innovative activities including program design, training, studies and implementation of pilot schemes. Although restructuring reduced construction program by a third, at the same time it expanded the quality improvement program to cover primary as well as middle schools. Component Base Cost Rating Improving Access 116,000,000 S Improving Quality 17,200,000 S Improving Organizational Capacity 3,700,000 U 3.5 Quality at Entry: Formal Quality at Entry Criteria were not in effect at the time this project was prepared but nonetheless, the project design anticipated many of the criteria now in place. In particular, the objectives of the Project are consistent with the Country Assistance Strategy presented to the Board on March 31, 1994. The CAS focused on the reduction of poverty and, in particular, stressed strengthening governance, improving the delivery of core social services and targeting women, children and the poor. The CAS highlighted the role of the Social Action Program as a vehicle for pursuing these ends. However, the appraisal document did not provide a detailed appraisal of the economic or social aspects of the Project. Moreover, the Ptoject's financial management system did not receive the in-depth attention at appraisal that more recent projects have received following the introduction of OP/BP 10.02. The appraisal of the Project did not include an assessment of the depth of commitment of the Borrower or of the readiness of the Project for implementation. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The Project has exceeded its restructured targets for increasing access to middle school education and has substantially met most of its targets (enlarged at restructuring to include primary as well as middle schools) for improving the quality of schooling. More than 400,000 additional student places will become available upon completion and final approval of all schools constructed, upgraded or improved under the Project. A survey of schools sites, conducted in May 2000, suggests that the location of these schools meet agreed criteria and students are likely to attend. Due to delays in the approval of completed construction and recurrent budgeting for new schools, only about half of these facilities were in use as of the end of May 2000. The impact of completed schools on enrollment of girls cannot be rigorously determined. The Government conducted a school census in 2000 which revealed that enrollment of girls in middle schools - 3 - had increased by 27% since the beginning of the Project, and that boys enrollment had risen by about 3% over the same period. However, this growth cannot be attributed to the Project alone; disentangling the effects of other developments from those of the Project would require an assessment of the counterfactual and this has not been possible. It is known that between 1992 and 2000 the net enrollment rate for primary schools declined from 53% to 44%, indicating that the growth in girls enrollment occurred against a backdrop of falling participation in elementary education. The Project has contributed significantly to improvement of the quality of education through the development of testing instruments, instruction of staff in modem methods of testing and assessment, and institutionalization of education evaluation. The Project has exceeded its initial and restructured goals for training staff in modem methods of teaching; school administration; and the teaching of mathematics, English, and science. The Project has provided slightly more than half of benefiting schools with improved science teaching resources and has developed and introduced supplementary reading materials into about 7500 primary and middle schools. The Project has also provided staff training and technical assistance to strengthen management information systems, reorganize the elementary education sector, and improve capacity for management of the sector. The Project's assistance program to develop School Management Conmmittees (SMCs) has also been noteworthy. Its efforts to persuade policy makers of the importance of quantitative policy analysis and of the value of quantitative approaches to education planning have been only partially successful, in part because of frequent transfers of senior staff. The Project's intention to improve the design, production and distribution of textbooks through the Provincial Department of Education, and transfer control and responsibility for elementary education to the Rawalpindi Municipal Authority were eliminated during restructuring in response to unresolvable interagency disputes concerning authority over these matters. 4.2 Outputs by components: Access andparticipation. Construction, expansion, renovation and furnishing of 2,428 schools were completed by the end of 1999. The renovation of an additional 562 schools was undertaken in 1999-2000; 446 of these schools were completed and a further 98 were under construction at the time the Project closed. Formal approval of recurrent budgets has been received for two-thirds of the new schools completed by May 2000. Nineteen percent of fully approved schools were not in operation in May 2000 because teachers had not yet been assigned to them. A survey of a sample of 153 schools constructed or improved under the Project revealed that the average attendance at new schools was 13.8 students per class while the average for those upgraded from primary to middle schools was 26.3. These enrollment figures are certain to increase as schools become better established and the school age population continues to grow. The scholarship program for disadvantaged girls, which was intended to encourage attendance in middle schools, was a mixed success. The criteria for distributing these scholarships were not defined prior to appraisal of the Project and considerable delays in identifying recipients occurred as a result. The first tranche of awards was made in 1996-97 when 62% of the planned number of scholarships were awarded. About 92% of scholarships were renewed in the second year, implying that nearly all recipients remained in school. The disappointing uptake during the first year was due in large part to the low nominal eligibility criteria (set at appraisal at a maximum family income of Rs. 10,000 a month), and very cumbersome arrangements for disbursing the funds (which had not been spelled out during project preparation). Scholarship winners were required to establish an account at a branch of the Muslim Commercial Bank which demanded that at least Rs. 500 be deposited in order to open an account. Many recipients were unable to make the initial deposit and thus were unable to utilize the awards. In 1997, the procedures for disbursement of scholarship funds were simplified by making post offices and other banking institutions acceptable disbursing agents and delivering the annual scholarship in two equal deposits. The scholarship program to encourage rural girls to pursue a career in middle school teaching has provided support to 51% of the targeted number of girls. Its full impact on girls' participation in middle school cannot be assessed until the scholarship winners have completed their studies, become fully qualified to teach, and actually - 4 - take up classroom positions. This process will require at least seven years. Raising the Quality of Middle School Education. Although this component of the Project did not achieve its goals during the early years of project implementation, very impressive strides were made following project restructuring in 1997 when a new project manager was appointed and greater focus on effective implementation was achieved. Lead trainers (one male and one female in each of 34 districts) were successfully trained in the use of testing and assessment materials. Eighty percent of school representatives trained by the Curriculum Research and Development Center (CRDC) report having implemented some of the new practices. The program to provide in-service training was expected to serve one teacher each in mathematics, English and science, and a head-teacher for each of 6000 schools. Mathematics and science teachers were provided two training sessions of two weeks duration and teachers of English were offered one, two-week session. Head-teachers were provided three, two-week sessions of training. Staff from 93% of middle schools participated in these training programs; 63% of teachers report that they have introduced practices learned from the courses into their classrooms. The two-week training program for English teachers has been acknowledged to have been too brief to have had a major long-term impact on classroom practice. The science education sub-component included the development of low-cost science demonstration materials, provision of training, and delivery of science demonstration benches. Science demonstration benches and other furniture were available and being used in more than 90% of the schools included in the survey of completed schools, but only half the schools had received the laboratory equipment and in fewer than a third of those schools was the equipment actually being used. Plans to produce textbooks at the provincial level that are both more attractive and better suited to the needs of middle school students were unsuccessful due to disputes between the province and the Federal Ministry of Education and its Federal Curriculum Wing, which claimed the authority to select all textbooks. As noted earlier, more careful examination of this complex relationship at the time of appraisal, might have resulted in more effective management of these complex relationships and interests. The initiative to provide supplementary reading materials was a major success. By the close of the Project, 219 titles had been produced and nearly 7,000,000 books had been distributed to 7500 schools. Training in the use of supplementary readers has been provided to teachers. In addition, through the Project, the private publishing industry discovered the existence of a large market for high quality children's books, both among private schools and middle class families. The country's two largest chains of private elementary schools--Beacon House and City Schools--have adopted the materials for use by their nearly 60,000 students. The DoE for Northwest Frontier Province also plans to introduce the supplementary readers in its schools. The testing and assessment component has also been recognized as a success. A Test Development Center (TDC) has been established with capacity to prepare test items, analyze test results and employ these results in the management of schools. Technical assistance to the TDC has been funded by DflD. A computer facility has also been created, but requires additional staffing to become fully effective. However, staff trained abroad in the preparation of test items and the analysis of test results have been reassigned to other agencies and non-specialists have been appointed in their places. On-the-job training has been provided to the new staff but a professional testing cadre will have to be established to ensure full implementation of a permanent, high quality testing and assessment program. Workshops and training courses have been provided to teachers in the drafting and evaluation of test items. An effort to provide the TDC with a secure facility for the development and storage of tests has not yet succeeded. - 5 - Strengthening policy-making, planning, management, monitoring and evaluation. The most notable accomplishment of this Project component has been separation of the administration of elementary and secondary education. The Project supported this initiative to ensure that greater attention is paid to the needs of the elementary sector. The civil service has been bifurcated down to the level of the Deputy District Education Officer and the management of the elementary schools have been given greater attention as a result. The Project also sought to improve policy-making and planning through the development of reliable data bases and the strengthening of analytic capacities. The school construction program was supposed to have been guided by a geographic information system which in turn was to have been informed by data gathered by the newly formed Education Management Information System (EMIS). The EMIS cell has been established but has not been provided with either the budget or staff necessary to carry out its mandate effectively. Supervision missions repeatedly brought the needs of EMIS to the attention of senior officials of the DoE, but the DoE has not yet acted on these recommendations. The lack of support for EMIS appears to reflect the low priority placed on the use of data in policy analysis by the DoE. The School Mapping Information System has not been implemented. The Project also promoted the creation of an interagency coordination body--the Education Coordination Committee--which was intended to link efforts by the agencies responsible for curriculum development, testing, staff development, textbook distribution, construction supervision, and data collection and analysis. During the restructuring of the project, it was agreed to create an Education Development Unit in the Planning and Coordination Wing of DoE. This unit has not been adequately staffed in spite of repeated prodding by supervision missions. As a result the EDU has not yet been able to play a serious role in policy making or planning. Finally, the restructured Project afforded support to SMCs. SMCs were introduced under the Primary Education Project and taken up by the Social Action Program as devices for ensuring greater community participation in the govemance of schools, and for mobilizing local resources for education. The Project has provided funds to enable non governmental organizations (NGOs) to conduct training in financial management and community mobilization for members of the SMCs. Training materials have been prepared and day-long seminars have been provided to SMCs. The SMCs provide both community oversight of schools and help to mobilize community resources for education. The original design of the project did not include an evaluation strategy beyond normal supervision reports. DflD supported evaluation studies for the activities it funded which contributed to an understanding of their impact. However, in the absence of a more fully elaborated program of evaluation, the short-term impact of the measures undertaken to improve the efficiency and equity of resource use in middle schools can not be rigorously measured. 4.3 Net Present Value/Economic rate of return: The economics of this project were not analyzed during preparation or appraisal. 4.4 Financial rate of return: The financial implications of the project have not been formally considered. 4.5 Institutional development impact: This Project has launched a large number of innovations and has had a pronounced impact on the thinking of agencies concemed with management information, testing, materials development, teacher training, - 6 - gender and community participation. In terms of its impact on the values and aspirations of these agencies, the Project has been highly effective. From the more narrow perspective of organizational development, the Project has been less successful. The attempt to increase reliance on rigorous policy analysis has not yet been fully successful. The endemic problem of rapid turnover of senior staff continues. The DoE remains unable to articulate a clear vision for education in the Punjab or to mobilize broad support for its policies and plans. The data collection and analysis functions continue to be under-funded and under-staffed which undercuts their capacity to ensure the policy relevance of statistical reports. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control ofgovernment or implementing agency: During the early years of project implementation, frequent political changes adversely affected staffing and resulted in a lack of senior managerial attention. The suspension of assistance from DflD following the change of government on October 12, 1999 did not significantly delay implementation of the Project, but did affect the ability of the Government to undertake a final assessment of the full range of quality components. 5.2 Factors generally subject to government control: Frequent changes in the leadership of the DoE interrupted progress and contributed to shifts in the focus of the Project. (During the life of the Project, the average term of service was thirteen months for Secretary and seven months for Deputy Secretary.) Delays in the appointment of senior staff to the DoE and project entities led to the Project being placed in problem project status in April 1995. The pace of construction and renovation of schools was disappointing during the early years of project implementation first, because the criteria agreed at appraisal for selecting sites for the expansion or construction of schools were not sufficient in practice to identify sufficient numbers of acceptable sites; and second, because disputes between the DoE's Civil Engineering Cell (CEC) and the Communications and Works Department (C&W) over authority and capacity to manage construction activities were not resolved. The ability of the CEC to manage a large construction program was compromised by the unwillingness of the C&W to second project engineers as initially expected. The long period of uncertainty about the future of the CEC, and its eventual closure, crippled its recruitment of staff and adversely affected relationships with civil works contractors. A delay of more than four years in the construction and commissioning of the Test Development Center's new building frustrated the TDC's efforts. Conflicts between the Punjab Textbook Board and the Federal Curriculum Wing over the authority to select textbooks led finally to abandonment of efforts to improve textbooks. Cumbersome Government procedures for the selection of consultants delayed their appointment and reduced their effectiveness. 5.3 Factors generally subject to implementing agency control: Delays in the appointment of staff to agencies established to improve training, testing and production of textbooks and supplementary reading materials slowed the implementation of these components. The Government's commitment to the Bank to appoint a systems analyst to the staff of the EMIS (first reported in the Aide Memoire for the supervision mission conducted in December 1996) was never fulfilled. The replacement of five contract staff of the Test Development Center (TDC)--who had been trained in the United Kingdom under the Overseas Development Administration (ODA)/Department for International Development (DflD) technical assistance--with regularly appointed generalist civil servants reduced the impact of the investment in training and interrupted the development of the testing and assessment sub-component for at least six months. Completion of construction of a purpose-built facility for the TDC was delayed by nearly three years, thereby reducing the contributions that TDC was able to make during the project period. A delay of more than nine months in the approval of an agreement with the ODA for the provision of technical assistance slowed the development of teacher training, textbooks, teaching materials and assistance in the reform of education administration. The Project suffered because of weak financial - 7 - management and inadequate financial controls (as pointed out in qualified audit reports); delays in delivering audit reports; and unsatisfactory follow-up of significant audit observations. 5.4 Costs and financing: The timely release of funds from the ADP was a problem, particularly before the Project was restructured. Although adequate provision was made in the ADP, funds were released only after delays of three to six months. In addition, transfers of funds to the account of the CEC were delayed following the release of the ADP, leading to late payments to private contractors. Finally, the amount of the Special Account held by the Project Management Unit was not increased following the transfer of responsibility to it for payment of contractors during restructuring. This further delayed payments to contractors until the monetary ceiling of the Special Account was increased in October 1999 to reflect the new responsibilities of the PMU. During the restructuring of the Project, targets for the renovation of schools and the construction of new classrooms and schools were reduced by a third, funding for quality enhancing activities was increased by US$3.6 million and US$13.0 million of the Credit was canceled. A further US$30.0 million is projected to be unused at the close of the Project as a result of the Government's reluctance to cancel additional amounts of the credit. Approximately 57.5% of the original credit amount is expected finally to be disbursed. The PMU was unable to develop a system for transferring responsibility for management of the financing of civil works to the field offices, which slowed the implementation of this component. Payments that included retention monies deducted from the contractors bills were released either from Government of Punjab or IDA Special Account. IDA has started the process of recovery of its share of retention monies released to the field offices, and has advised the PMU to obtain details of actually released retention monies from the field offices and to claim retention monies that became due on or before the Credit closing date of June 30, 2000. 6. Sustainability 6.1 Rationale for sustainability rating: The likelihood that the Project will be sustained over the long run is considered very good. Institutional. The most central elements of the Project appear to be firmly established and to have substantial professional and political support. The PMSP has contributed significantly to the creation of capacity for: (a) testing and assessing student leaming; (b) preparing and producing teaching materials; (c) training teachers and headmasters; (d) providing scholarships and incentives to increase access to quality middle schooling; and (e) promoting community participation in school management. The TDC, the CRDC (which prepares and produces teaching materials), the Directorate of Staff Development (which provides teacher training) and the Directorate of Public Instruction-Elementary Education (which supervises primary and middle schools) have all demonstrated their worth. A culture of program evaluation and school improvement has emerged under the Project that enhances broad professional and public support. The program to supply Supplementary Reading Materials has built a strong partnership with the private publishing industry, increasing the likelihood of their continued use. However, the value of the scholarship and incentives programs has not yet been demonstrated and problems in the equitable administration of these programs have not been entirely overcome. The sustainability of efforts to increase community participation and strengthen accountability for education is uncertain due to the Government's on-going program of devolution of responsibility for social services to the district level. However, this shift in responsibility is expected to strengthen accountability to clients and to increase local participation in the governance of schools. The effectiveness of the devolution policy in achieving these ends will hinge on detailed plans for transferring responsibilities to local communities; these plans have not yet been fully elaborated. While the EMIS is functional, it has not yet been adequately funded or staffed, and has been unable to fulfill its mandate. The DoE has not yet begun to rely on fact-based policy analysis or systems - 8 - planning. FinanciaL The annual recurrent expenditures necessary to operate the programs established under the PMSP would be approximately 6% of the recurrent budget of the DoE before the Project was implemented. This sum is affordable within modest scenarios for public revenues and spending. The transfer of responsibility for education to the district level is expected to result in greater mobilization of local resources for education, and thereby further increase the affordability of improvements in schooling. The testing, training, material development, and community support components represent only about a quarter of the total incremental costs; these programs have vigorous champions and are likely to be adequately funded. The EMIS and the scholarship programs are less successful and, despite their importance (especially the EMIS), may not receive sufficient budgetary support to enable them to carry on effectively. Adequate budgets have been provided for teacher training and supplementary reading materials this year. 6.2 Transition arrangement to regular operations: The agencies responsible for testing and assessment, preparation and production of teaching materials, and training of teachers and headmasters are well rooted in the formal bureaucracy. The effort to rationalize teaching loads by reassigning teachers and supervisors from over-resourced to under-resourced schools has slowed the staffing of new and expanded schools. The provisions for collection and analysis of management information remain fragile. The scholarship program has not yet successfully implemented credible criteria for the granting of awards. Statements of New Expenditure have been adopted for nearly half of those schools that have been given approval for inclusion in the recurrent budget by the Planning Commission. The development of SMCs has been stymied by legal challenges to their authority to spend money from head teachers and local government officials. The recent reforn of the SMCs is expected to overcome these problems. The announced program to devolve responsibility for elementary education to district governments (which have yet to be formed) has contributed to uncertainty about the future govemance of the sector. 7. Bank and Borrower Performance Bank 7.1 Lending: The design of this project built on experience gained from the Bank's financing of nine education projects in Pakistan between 1964 and 1990, as well as from analyses of the implementation of a pilot project funded by USAID in Balochistan. These lessons were identified at the time of appraisal as follows: (a) project administration should be decentralized; (b) projects should be limited in scope to a single province; (c) new and expanded functions should be performed by existing bodies rather than new organizations; (d) staff skills and organizational capacity should be developed before complex changes are introduced; (e) goals should be realistic; (f) changes should be introduced incrementally; (g) agreements about the need for technical assistance and on terms of reference should be reached with government early on; (h) construction activities should be intensively monitored and supervised; (i) implementation mechanisms should be established early to ensure timely disbursements; and (j) programming should be flexible to allow adaptation during implementation. Review of the Project reveals that several of these lessons were not heeded. Implementing agencies have been shown to have great difficulty carrying out large programs of construction. The Project has also revealed that Government cannot bring about fundamental reforms quickly. The provisions for field supervision of site selection, construction and improvement of schools were inadequate initially; carrying out these functions in a province with more than 70 million people did not succeed until a fornal management structure was established at divisional and district level. The management of construction proved to be a major problem. The CEC was initially entrusted with this task but it lacked the experience -9- or capacity to implement such a large investment program. The assumption that it would be able to recruit engineering staff from the C&W proved to be false. The jurisdictional disputes that followed substantially delayed construction. The arrangements for site selection and construction supervision proved to be inadequate and had to be changed during implementation of the Project. The coordination of technical assistance from consultants financed by the Project with that supplied by DffD proved to be difficult during the early years of the Project. The unwillingness of the Government of Punjab and the DoE to employ consultants became an issue during project implementation. Monitoring and evaluation activities being carried out by DflD and by DoE suggest that some of the Project's components may not yield all the benefits projected during appraisal. For example, investments in the renovation of schools and the construction of new classrooms and schools have not yet fully led to substantial increases in enrollment primarily due to delays in implementing the civil works components of the Project, but even where schools have been constructed and put into service, the increase in enrollment has sometimes been smaller than predicted. This suggests that the constraints on the enrollment of girls in middle schools are more complex than initially assumed. Due to initial problems in the selection of awardees and the disbursement of funds, the provision of scholarships to girls eligible to attend middle schools can not be linked in a cause and effect manner to increased participation. DflD evaluation studies suggest that the brief periods of teacher and headmaster training may not achieve lasting behavioral change, especially in situations in which the other requisites for improvement of teaching and learning may be unmet. The initial project appraisal lacked an assessment of the existing financial management systems and procurement capacities of the borrower. Had procurement actions been better specified in advance and procurement capacities been more carefully examined, and a procurement plan been formulated at that time, subsequent financial management difficulties might have been avoided. Similarly, the initial project documents lacked analyses of the economic costs, effects or benefits of the Project, or clear specification of the Project's expected effects. The Bank's performance improved sharply following the restructuring of the Project. The improvement appears to be primarily the result of greater continuity in the staffing of supervision missions and support provided between formal mission reviews. In addition, Government provided greater oversight and support for implementation following restructuring of the Project. 7.2 Supervision: Supervision of the Project was led by five different staff members over the first three years of project implementation but changed to a single staff member during the last four years. The aide memoires during the first four years raised broad issues and identified problems. After the Project was restructured, the team supervising the Project was kept intact and action plans were set forth in aide memoires. Financial management specialists were not involved in supervision until the last year of the Project, as these have only recently been engaged by the Bank. Better supervision of financial management would have led to more timely action in implementing proper record keeping, adequate internal controls, accurate financial statements, timely audit reports and appropriate follow-up of audit observations. While most procurement was ably carried out, problems in the procurement of civil works were discovered during the final year of the Project. These problems included providing too little time for bidders to respond to tenders. Efforts are currently underway to match claims for reimbursement with contracts and other records. A project launch workshop was proposed during the February 1993 supervision mission and the suggestion was repeated in seveTal subsequent aide memoires, but the workshop was not undertaken. - 10- 7.3 Overall Bankperformance: The overall performance of the Bank was marginally satisfactory. The Bank did not appraise the initial design of the Project and its implementation arrangements in sufficient depth to ensure its smooth implementation. This led to delays in the implementation of the Project and cancellation of a substantial fraction of the original credit amount. However, the Project performed well after it was restructured. Greater continuity of both Bank and DoE staffing and increased stress on targeted action plans improved follow through on important tasks. Providing a project launch workshop emphasizing procurement and disbursement procedures and accounting requirements might have enabled the DoE and PMU to comply more fully with the Bank's requirements. Borrower 7.4 Preparation: The Project was designed to contribute to the implementation of the province's Elementary Education Development Model. The Government of Punjab worked closely with the Bank in preparing and approving detailed working papers for the Project on the following topics: (a) deficiencies in existing elementary schools; (b) local government participation in elementary education; (c) private sector participation in elementary education; (d) enhancing teacher effectiveness; (e) reforming assessment of student achievement; (f) improving the textbooks and teachers' books; (g) supplementary materials collections in elementary schools; and (h) strengthening policy-making, planning, management and evaluation. 7.5 Government implementation performance: Very strict adherence to the initial project document approved by the Planning Commnission (the PC- 1) kept the Project from responding effectively to lessons learned during project implementation. During the pre-restructuring period, rapid tumover of senior staff in senior leadership roles in the DoE and as Project Director resulted in frequent shifts in direction and substantial periods of confusion as new appointees struggled to understand their jobs. Delays in the release of funds from the ADP for up to seven months over each annual budget cycle, and the withholding of authority to spend released funds, adversely affected the completion of civil works and eroded confidence in the CEC. The lack of cooperation between the Federal Curriculum Wing and the textbook component of the Project led to the cancellation of this sub-component. Failure to appoint systems analysts to the staff of the EMIS undercut the organization's ability to produce policy relevant information and thereby build its credibility with policy makers. Requiring that all consultants funded through DfID technical assistance be vetted by Government slowed the selection and appointment of important technical advisors. Project implementation turned around from 1997 onwards, and peaked in 1998/99 as a result of greater staff continuity, provision of greater authority to the project director, stronger leadership by the Department, and high level regular oversight by senior Government officials and by the political leadership of the province. 7.6 Implementing Agency: Uncertainty over whether CEC or C&W would implement the civil works program hampered implementation of the Project for nearly three years. In addition, uncertainties surrounding the future of CEC led to low staff morale and hesitance among contractors to complete tasks. Failure to appoint persons trained abroad in testing and evaluation to permanent posts under the Project deprived the TDC of valuable skills. Supervision of procurement was confined to the project office, and focused mainly on the procurement of goods and services. Adequate training and supervision were not provided to the C&W Department, or to districts to ensure that agreed procedures were being followed. The lack of understanding by some districts led to serious deviations from acceptable procedures which could lead to declarations of misprocurement. The value of international technical assistance would have been greater if government - 11 - counterparts had been identified early and had worked more closely with international staff. Efforts to cooperate with private publishers in producing supplementary reading materials contributed to the emergence of a vigorous private publishing industry for children's books. Delays in appointing staff to the EDU undermined its effectiveness. Failure to appoint analysts to the EMIS staff reduced the capacity of the cell to deliver policy-relevant data and analyses. Delays in the appointment of DfID consultants resulted in there being too little time for providing valuable support in implementing reforms. The project management team did not implement formal accounting procedures to ensure an effective system of internal control despite repeated reporting by the auditors. Adequate project accounts and records were not maintained. Figures per the accounting records and those used by Project Management (provided by the CMU) for reporting, remain un-reconciled. The reported figures for Civil Works are based on releases made and are not fully reconciled with actual expenditure incurred. Project asset records are also inadequate and there is no evidence of confirmation through a physical stock check of assets compared with recorded values. In some cases audit reports were not in full compliance with IDA guidelines and issues raised by the auditors could have been addressed in a more timely manner. Based on the audit observations, certain "ineligible" expenditures have been incurred (which may lead to possible recovery of funds). Audited accounts of the CEC and CRDC are not available with the PD. The lack of proper financial management/accounting has resulted in incorrect claims made to the Bank, which include unpaid retention money. Adjustment will therefore be required through subsequent claims prior to final disbursement. Several financial management issues were raised during the supervision missions at the end of the Project. These have remained largely unresolved because of lack of appropriate capacity of the staff in the Project Director's office. 7.7 Overall Borrower performance: During the last three years, staff continuity and high-level oversight contributed to the Project coming close to accomplishing most of its restructured targets. This positive result followed an initial period when frequent turnover among key staff of the Project and awkward implementation arrangements delayed progress. The progress in implementation was a result of the sustained attention to ifnplementation during this period. Lack of training for project-related government staff in procedures and reporting requirements of the Bank led to serious procurement and disbursement problems, which may result in some expenditures not being eligible for financing from the credit. 8. Lessons Learned From preparation to appraisal stage: * Stakeholders should be consulted extensively during the design of a project. The effectiveness of training efforts, in particular, would have been much greater had the DoE participated more fully in their design. * The contents of a project should be fully integrated into the government's strategic plan for the sector. Linkages between the PMSP and the Social Action Program were not fully understood by stakeholders, particularly those at the district level, even though the Social Action Program had been adopted as the overarching policy umbrella for elementary education. * The size and scope of a project should be determined to be administratively feasible. The Project was initially intended to be implemented by the DPI-E. A separate PMU was created during implementation to provide greater administrative capacity. Even then, the Project proved to be too ambitious to be managed effectively. Smaller, more focused projects that pursued a narrower set of objectives (such as teacher training/development, or examinations reform, materials development) and relied on a single - 12 - specialized agency (such as the DSD, or CRDC) might have stretched staff less thinly and been more successful. * The capacities of implementing institutions should be carefully assessed and their responsibilities should then be clearly defined during project preparation. The CEC was unprepared for the responsibilities assigned to it and the DPI-E and PMU were not adequately prepared to meet World Bank reporting requirements. * Project components should befully elaborated and appraisedfor their technicalfeasibility by both government and donors during project preparation. The robustness of proposals to possible institutional deficiencies, and administrative and bureaucratic risks should be evaluated in advance. Plans for the implementation of innovative components should be given particular attention. For example, the scholarship program for disadvantaged girls encountered difficulties during its implementation because the criteria for awarding scholarships had not been fully elaborated and arrangements for the disbursement of awards had not been fully thought through. * The needfor communication among government entities should be assessed during preparation and arrangements should be made to ensure that routine exchanges of information occur throughout the life of the project. Especially during the first four years of the Project, contacts within and between departments were insufficient to ensure appropriate follow through on issues and resulted in unnecessary delays. For exarnple, despite repeatedly being informed that the Government of Punjab would be liable for the cost of completing unfinished project activities (such as civil works, etc.), the implications of this were not fully grasped. * A detailed appraisal of the borrower's financial management system should be carried out and essential improvements should be introduced prior to effectiveness and disbursement of the Project. The Bank's reporting requirements should have been communicated in detail to the implementing agency/borrower. e The roles and responsibilities of implementing agencies should be clearly defined during project preparation. Planning and Development, Finance, and DoE not only should have been apprised of their respective responsibilities, but should also have reached agreement on their mutual commitments. * The responsibilities of key staff should be clarified before implementation of a project. In several instances the detailed objectives of Project components and corresponding terms of reference for the staff responsible for carrying out those activities were not clear until mid project. Staff often found themselves held accountable for meeting objectives that had not been clearly articulated and for satisfying procedures that had not been made available to them, (including those described in Bank documents (SAR and Project Agreement)). During Project implementation: * A project launch workshop should be held to familiarize staff responsible for procurement, disbursement andfinancial management with Bank procedures and requirements. District level staff responsible for contracting for civil works should be encouraged to participate in the workshop. * High-level oversight and continuity of staffing should be ensured by Government. * The feasibility of the scope and size of the project should be very realistically appraised before the legal agreement is signed since it is very difficult to reduce the project size and scope during - 13 - implementation. * The Bank's technical input and assistance should be made available morefrequently and systematically. The Bank should seek to work with officials other than those in the Secretariat. * Meetings between project management and specialized agencies (such as DSD, CRDC and EMIS) should be held regularly to ensure good communication and close coordination. * Continuity should be maintained among key project personnel. Continuity is especially important for innovative project components. Changes in key personnel have resulted in a loss of expertise and momenturn and have resulted in a waste of approximately three months of productivity per major transfer. * Restructuring can delay implementation. Between six and twelve months of implementation time was effectively lost when the project was restructured. * Consultants should be chosen who are conversant with local conditions; employment of consultants with divergent views should be avoided. The impact of short-term technical assistance has been unsatisfactory, particularly in the areas of policy-making, planning and management. * A champion should be identified who would ensure that reports from consultants are reviewed by senior government officials, that any recommendations for action are properly evaluated and that a course of action is agreed upon. * Formal Bank supervision missions, backed by continuous, on-going interaction with project staff were important in resolving emerging issues. * Teacher training should befollowed by a continuous monitoring and support to teachers. A wholesale transfornation of approaches to classroom learning by teachers up through district level supervisors is required to ensure effective practice. * Teachers should be recognized and rewardedfor innovations in teaching behavior. The weight of tradition and conservative supervision discourages departures from established teaching norms. * Successful implementation of a project requires broad and sustained supportfor its objectives, not only from Government but also from major stakeholders, and an implementation team that is both competent and supported by policy makers. 9. Partner Comments (a) Borrower/implementing agency: The Project encountered significant delays in implementation due to administrative problems. The DPI-E initially was intended to implement the Project and the PIU was to have monitored progress, but instead implementation was carried out by the PIU with the DPI-E operating the accounts only. In 1996, the Project Manager was re-designated as Project Director and the DPI-E began functioning as Coordinator. An effort to obtain the services of a private finn to supervise construction failed for technical reasons, delaying implementation of the civil works component. Subsequent closure of the CEC and transfer of its responsibility for construction work to the C&W Department further delayed implementation. In addition, responsibility for the procurement of school furniture and science demonstration desks and equipment had - 14- to be assumed by the PIU following a legal challenge to delegation of this responsibility to SMCs. Additional major reasons for delay in implementation of the Project are (a) frequent changes of the Project Directors/Project Managers; (b) failure of PIU staff to complete imnplementation and monitoring tasks; (c) insufficient participation by DPI-E in implementation/monitoring work; (d) changes in policy regarding implementation of civil works, procurement and scholarship/incentives; (e) delays due to restructuring of PC-I and introduction of new activities; and (f) reduction of the actual implementation period to only two years (for the reasons cited earlier). Implementation of the civil works program prior to restructuring was disappointing. However, following the reassignment of responsibility for construction to the C&W Department, targets were more than met. The Government of Punjab proposed extension of the Project period in order to allow it to make up for the time lost during early project imnplementation and thus to utilize the savings and unallocated funds, but the World Bank would not agree. All of the other components of the Project were also implemented in one form or the other. However, the Project did not strengthen EMIS appropriately despite providing about 70 computers and other accessories. Similarly, the EDU did not really take off. The EMIS could not be improved as its technical capacity was not developed. The EDU was created but did not receive proper attention and its potential was not realized, primarily because execution of project components did not depend upon the EDU and thus the latter fell into neglect. The SRMs program needs to be further evaluated. There is no clear evidence that the SRMs have caught the popular imagination. The ICR rightly observes that the impact of the Project and of individual project components can not be rigorously assessed. This was so as no formal attempt was been made to carry out complete or sample survey study. Most of the quality components being carried out by the Project have now been taken up by the DoE and are being funded through regular budget of the Department. These include (a) shifting EDU positions from development to non-development; (b) providing sufficient allocation to EMIS to enable it to carry out its activities effectively; (c) training teachers and head teachers in the subjects of Mathematics, English and Science (MESH); (d) providing scholarships to girl students to enhance enrollment and retention; (e) offering cash incentive to teachers; and (f) training of teachers in admniristration and in working with SRMs. The lessons learned, as depicted in the ICR, will be helpful to the DOE/GOP and the Bank in designing and implementing new projects in the future. One such lesson is regarding the project period. Initially, the Project was slow, it picked up gradually, especially after its restructuring in 1997. The project performance peaked only in the last two/three years of its life. No extension was provided. Had the project period been extended for a year or two, it would have been possible to better capitalize on the Project successes/initiatives - leading to a more sustained impact of the quality components. (Thefull, unedited text of the Government's Comments is presented in Annex 8.) (b) Cofinanciers: DFID. DFID agreed to fund technical assistance, through a grant of $6.3 million, for teacher and headteacher training, development of textbooks and supplementary readers and development of assessment and testing. Particular attention was given to measuring impact. Gender and other equity issues were mainstreamed. At the request of the government, DFID's original period of assistance was extended until June 2000. Despite some restrictions on assistance during the project period, DFID provided significantly more support than had been anticipated originally in the SAR design. Overall, DFID is satisfied with - 15- project outcomes and with the performance of the consultants it funded. The Project was particularly successful in developing high quality supplementary readers and developing the capacity of the private publishing sector. Future sustainability will however require a system for school purchase of new and replacement titles, further development of editorial, technical and marketing expertise and a stronger partnership between the publishers, the Punjab Textbook Board and the Punjab Education Departrnent. Training of teachers and headteachers would have been more fully sustained in terms of changed practice in schools if DEOs and DDEOs had been more involved. Although attention was given to monitoring and evaluation, capacity still needs to be institutionalized. The Project worked best when there was effective leadership and good co-ordination at all levels of project implementation. Although Project outcomes were ambitious, and were subject to the institutional and capacity constraints identified in the ICR, they are broadly in line with experience elsewhere in Pakistan. We concur with the general conclusions of the ICR, but believe that the key lessons are not so much about getting things right before the project starts, but about establishing a process based on partnership, supported by financial and technical resources flexible enough to respond to development needs as they arise. (c) Other partners (NGOs/private sector): 10. Additional Information The Government of Punjab requested an extension of the Project to allow it to renovate or construct additional facilities. The Bank, citing Regional policy, has declined to extend the closing date for the Credit. The failure of district authorities to comply with Bank procurement and disbursement procedures may result in IDA not being able to disburse against some expenditures. - 16 - Annex 1. Key Performance Indicators/Log Frame Matrix Outcome I Impact Indicators: _twbx Pt n bstPS Actua4LxttEstimEat Increase in gross enrollment rates Gross enrollment rates increased by 10.87% The increase in girls' enrollment is estimated for females and 3.95% for males between to be about half as large as projected at 1995S96 and 1997/98 according to PEMIS appraisal, but does not take into account the reports, the last school census available. enrollment in last two years of the Project or project enrollment in immediate post-project period. Improve the quality of education Output indicators; ., . I dI n .o IMatvlx Pr*ct.d In bit P,, tEovaJiLFWstftat. Construction of new elementary schools 236 new schools have been constructed and cleared by inspectors Upgrading of primary schools to elementary 542 schools have been upgraded and schools inspected Renovation of existing elementary schools 1074 schools have been renovated (An additfonal 1138 schools have been either completed or are under construction, but have not yet been cleared by inspectors) Training of tchers of English language, In 93% of middle schools, teachers math, science partidpated in training and 63% of classroom teachers report having introduced new teaching methods in the classroom as a result Training of head teachers in school Approximately 6000 head teachers management participated in six weeks of training activities Development and distrlbution of low cost A survey of 153 schools revealed that hatf science training materials to new or had received the inputs by May 2000 and a renovated schools third of those were using the new materials Produce and distribute supplementary More than 7,000,000 volumes have been reading materials to 7500 schools delivered to schools Separation of the administration of Separate Directorates of Public Instruction elementary and secondary educabon have been created for elementary and secondary education Creation of a Punjab EMIS EMIS has been established but is understaffed and underfunded. Creation of a Test Development Center Staff has been trained, a computer center established, a test item bank created and capacity for analysis of tests built up End of project - 17- Annex 2. Project Costs and Financing Project Cost by Component (in US$ million equivalent) Improved Access and Participation 116.70 82.84 70.99 Improved Quality 17.20 14.77 85.87 Strengthenened Management 3.70 0.00 0 Total Baseline Cost 137.60 97.61 Physical Contingencies 13.80 0.00 Price Contingencies 12.70 0.00 0 Total Project Costs 164.10 97.61 Total Financing Required 164.10 97.61 Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) 1. Works 0.00 103.00 4.10 0.00 107.10 (0.00) (78.30) (4.10) (0.00) (82.40) 2. Goods 0.00 12.80 9.80 0.00 22.60 ______________________ (0.00) (11.50) 8.10) (0.00) (19.60) 3. Services 0.00 0.00 6.20 0.00 6.20 Technical Assistance (0.00) (0.00) (6.20) (0.00) (6.20) 4. Miscellaneous 0.00 0.00 0.00 28.10 28.10 Incremental Recurrent (0.00) (0.00) (0.00) (6.90) (6.90) Costs 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 0.00 115.80 20.10 28.10 164.00 (0.00) (89.80) (18.40) (6.90) (115.10) Detailed figures may not add up to totals because of rounding. Figures in brackets are the respective amounts financed by the IDA Credit. - 18 - Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) Procumment Method' Efwc Cat9 O- 2 -- N.B.F. TOW Cost 1. Works 0.00 64.51 0.00 0.00 64.51 (0.00) (49.03) (0.00) (0.00) (49.03) 2. Goods 0.00 8.13 4.80 0.00 12.93 (0.00) (7.01) (4.38) (0.00) (11.39) 3. Services 0.00 0.00 9.46 0.00 9.46 Technical Assistance (0.00) (0.00) (4.34) (0.00) (4.34) 4. Miscellaneous 0.00 0.00 8.80 0.00 8.80 Incremental Recurrent (0.00) (0.00) (6.46) (0.00) (6.46) Costs 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 0.00 72.64 23.06 0.00 95.70 _ (0.00) (56.04) (15.18) (0.00) (71.22) I Figures in parenthesis are the amounts to be financed by the Bank Loan. All costs include contingencies. @ Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. Project Financing by Component (in US$ million equivalent) *k' -GeuL S-. *W;: 'G = Git g' Bb k G £DF Civil Works 78.30 24.70 0.00 48.69 18.14 0.00 62.2 73.4 0.0 Furniture, Equipment and 19.50 2.30 0.00 11.14 1.25 0.00 57.1 54.3 0.0 Materials Consultants and 10.30 0.00 0.00 3.88 0.12 6.90 37.7 0.0 0.0 Construction Management Services; Studies, Fellowships and Training Female Teacher and 5.50 5.30 0.00 1.65 0.70 0.00 30.0 13.2 0.0 Student Scholarships Vehicles 0.00 0.80 0.00 0.00 0.40 0.00 0.0 49.2 0.0 School Maintenance 0.00 7.30 0.00 0.00 3.59 0.00 0.0 49.2 0.0 Incremental Staff Salaries 1.40 8.70 0.00 0.04 0.82 0.00 2.9 9.4 0.0 and Operational Costs Support, Including Grants 0.00 0.00 0.00 0.29 0.00 0.00 0.0 0.0 0.0 for Non-Governmental organizations, School Management Committees, Community Based Organizations, and Pilot Innovative Activities Total 115.00 49.10 0.00 65.69 25.02 6.90 57.1 51.0 0.0 1/ An amount of USD 15 million was cancelled on September 3, 1997. The revised value of Credit comes to USD 100 million. Therefore, 65.7% of the revised Credit value has been disbursed. - 19- Annex 3: Economic Costs and Benefits Economic costs and benefits were not analyzed at the time of project appraisal nor at the time of the ICR. - 20 - Annex 4. Bank Inputs () Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e..& 2 Economists. I FMS, etc.) Implementation Development Month/Year i Count Specialty Progress Objective Identification/Preparation Appraisal/Negotiation June 28-July 18, 1991 4 Two education specialists, one disbursement officer, one architect consultant Supervision February 1993 5 Two economists, one S HS architect, one textbook specialist, one implementation specialist April 1993 3 Two implementation specialists, S HS one architect September 1993 4 One textbook specialist, one S HS education specialist, one implementation specialist, one architect April 1994 5 One implementation specialist, S S one architect, one financial analyst, one economist, one education specialist November 1994 3 One education specialist, one S S implementation specialist, one economist May 1995 5 One economist, two education U U specialists, one architect October 1995 4 Two education specialists, one U U economist, one architect April 1996 3 One education specialist, one U U management specialist, one architect January 1997 4 One education specialist, one S S senior social sector specialist, one economist, one architect December 1997 6 One senior social sector U S specialist, two education specialists, one economist, one architect, one team assistant May 1998 4 One senior social sector U S specialist, one education specialist, one architect, one team assistant - 21 - November 1998 5 One senior social sector S S specialist, one education specialist, one community development specialist, one architect, one team assistant July 1999 6 One senior social sector S S specialist, two education specialists, one architect, one financial management specialist, one team assistant January 2000 9 One senior social sector S S specialist, one disbursement specialist, one procurement specialist, one social development specialist, one education specialist, one information technology specialist, one architect, one consultant (operations), one financial management specialist ICR July 2000 9 One principal economist/task leader, one senior social sector specialist, two senior education specialists, one operations officer, one social development specialist, one disbursement analyst, one procurement analyst, one financial management consultant -22 - (b) Staff. Stage of Project Cycle Actual/Latest Estimate I_____________________ _ g No. Staff weeks s (,000) Identification/Preparation ___ Appraisal/Negotiation 148.90 414,746.00 Supervision 159.00 373,103.27 ICR 23.06 74,339.79 Total 330.96 862,189.06 (c) Trust Fund: Stage of Project Cycle c st Estite .j. No. of Staff weeks I USS (.000 Preparation 5.0 12.177.00 Supervision 14.5 38,874.00 Total 19.5 51,051.00 -23 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating O Afacro policies 0 H O SU0M 0 N 0 NA C, Sector Policies 0 H OSUOM O N * NA El Physical O H *SUOM O N O NA O Financial O H OSUOM O N * NA a Institutional Development 0 H 0 SU * M 0 N 0 NA El Environmental OH OSUOM O N * NA Social El Poverty Reduction O H OSU*M O N O NA El Gender O H *SUOM O N O NA O Other (Please specify) 0 H OSUOM O N * NA OII Private sector development 0 H 0 SU 0 M 0 N 0 NA El Public sector management 0 H O SU 0 M 0 N 0 NA El Other (Please specify) 0 H 0 SU 0 M 0 N 0 NA - 24 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating O Lending OHS OS *U OHU O Supervision OHS *S OU OHU O Overall OHS OS O U O HU 6.2 Borrowerperformance Rating E Preparation OHS OS * u O HU O Government implementation performance O HS * s 0 U 0 HU O Implementation agencyperformance OHS OS * u O HU O Overall OHS OS Ou O HU - 25 - Annex 7. List of Supporting Documents Staff Appraisal Report. World Bank. March 20, 1992. Development Credit Agreement (Punjab Middle Schooling Project) between Islamic Republic of Pakistan and International Development Association, Credit Number 2354-PAK. World Bank. June 5, 1992. Project Agreement (Punjab Middle Schooling Project) between International Development Association and Province of Punjab. June 5, 1992. Project Status Reports (PSR Nos. 1- 18). 1992-2000. Aide Memoires of Appraisal and Supervision Missions. World Bank. 1992-2000. Punjab Education Management Information System Census Reports. Government of Punjab, EMIS Cell. 1993-1998. Amendment of Development Credit Agreement and Project Agreement. World Bank. September 3, 1997. Correspondance During Restructuring of PMSP and Aide Memoire. World Bank. 1997. Strengthening the delivery of Education. Cambridge Education Consultants. May 1998. Second Social Action Program Project (SAPP II). Multi-Donor Review Mission, National Overview Aide Memoire. World Bank. February 2000. Second Social Action Program Project (SAPP II). Multi-Donor Draft Operational Plan Review, Discussion Notes, Punjab. World Bank. April 2000. Second Social Action Program Project (SAPP II). Multi-Donor Review Mission, National Overview Aide Memoire. World Barik. June 2000. Project Completion Report: PMSP ("DFID PCR '". Second Draft. Dr. Richard White, Senior technical Adviser, DFID. June 2000. A Report for preparation of the ICR for the Punjab Middle Schooling Project: Punjab Middle Schools Survey. EDC. August 2, 2000. Punjab Middle Schools Survey (May-June 2000). Enterprise & Development Consultants (Pvt) Ltd. August 2000. Impact Evaluation of Supplementary Readers. Dr. Muhammad Anwar, Director, Social Sciences Research Centre, University of the Punjab, Lahore. 2000. Research and Evaluation Series, 2 - 10: PMSP: "Enabling student learning through supplementary readers in Punjab elementary schools". DFID. June 2000. "Improving training strategies: Piloting demand-led training in the Punjab". DFID. June 2000. - 26 - "Changing school cultures: Follow-up case studies of Punjab Middle Schools". DFID. June 2000. "Impact of supplementary readers on Urdu reading comprehension of Middle grade students in Punjab". DFID. March 2000. "Evaluating the impact of in-service training: Case studies of Punjab Middle Schools". DFID. March 2000. "In-service teacher training: Assessing effectiveness through stakeholder impact monitoring". DFID. March 2000. "A baseline survey of learner achievement in grade 8: A summary of key findings". DFID. July 1999. "The use of matrix ranking to monitor the effectiveness of in-service training". DFID. July 1999. "A study into the use of supplementary readers". DFID. December 1998. -27 - Annex 8. Comments of the Department of Education on the Punjab Middle Schooling Project (EDA-2354) 1. The ICR is comprehensive and covers the objectives, components, constraints in implementation, achievements, and lessons learned regarding the PMSP. The Project started in 1992. It was to be implemented by the DPI (EE) and PIU was to monitor the progress, but its implementation was practically carried out by PIU and the DPI(EE) operated the accounts only. During 1996, the post of Project Manager was re-designated as Project Director and the DPI(EE) started working as Coordinator. It was observed that the construction work was quite slow, therefore, in consultation with the World Bank it was decided to acquire the services of the Construction Management Firm (CMF) from the private sector to ensure the implementation of school building construction. The bidding documents were prepared and technical and financial proposals invited through NCB. These were scrutinized/evaluated but the work could not be assigned to CMF due to certain technical reasons. It took quite some time to close CEC and transfer responsibility for construction work to the C&W Department. Besides, in case of procurement of school furniture, it was agreed to entrust this work to SMCs to involve the community in the improvement of elementary education and schools. But the procurement could not be entrusted to SMCs. Instead PIU had to develop bidding documents and procure the school furniture and science demonstration desks and equipment as per World Bank procurement procedure. Due to unsatisfactory achievement of the Project till then, its restructuring was proposed by the multi-donor mission. The restructured PC- I was approved by the ECNEC in 1997. The Govt. of the Punjab proposed extension of the Project period to utilize the savings, and unallocated funds but the World Bank could not agree. However, the additional construction work proposed was taken up and mostly completed. 2. Impact evaluation has not been formally carried out, making assessment of goal attainments nebulous, and the civil works component was a looming presence, an end in itself, overshadowing seemingly afterthoughts of quality education etc. The project was able to construct 2,989 schools (including 216 under construction), the DSD hostel, Test Development Center in the CRDC, and 11 GCETs. The other mentionable achievements are supply of furniture to 2,429 schools and production of SRMs. 3. Construction of new schools and adding classrooms to existing ones were to increase seating capacity (enrollment). Enrollment aspect has been only partially successful: the inability or unwillingness of parents to send their children to school has other dimensions peculiar to the socio-economic conditions of countries like Pakistan. 4. The project components, other than the civil works and furniture supply, were imaginative. The ICR terms the SRMs a success but it is felt that this needs to be further evaluated. It speaks of the publishing industry and private schools (Beacon House and City School) having benefited from this initiative. There may have been initial sporadic bursts of enthusiasm about the SRMs but finally there is dissipation with no study available to suggest whether the SRMs overwhelmingly and institutionally caught the popular imagination. (Beacon House and City School do not really require the SRMs as the textbooks current with them encompass the SRM material and the students of those schools, generally speaking, come from that strata of society where the contents of the SRMS are not unknown.) EMIS could not be improved as its technical capacity was not developed. EDU was established but could not receive proper attention or its potential realized. This was due to the fact that the execution of project components did not depend upon the EDU and thus the latter fell into neglect. 28 5. Since the start of the Project in 1993, it has been facing problems due to which its progress was initially very slow. Major causes for the slow progress are as under: (i) Changes of the Project Directors/Project Managers. (ii) Implementation and monitoring only by PIU staff which was never completed. (iii) DPI (EE) did not share the implementation/monitoring work. (iv) Policy changes for implementation of civil works, procurement and scholarship/incentives. (v) Restructuring of PC-1 and inclusion of new activities. (vi) The implementation period was practically only two years. 6. Construction of school buildings was the major component. The progress of the civil works remained slow due to the following reasons: (i) CEC was not well equipped to take up the works and accomplish the same as per schedule. (ii) Efforts were made to engage private sector for the implementation of civil works but the work could not be assigned to Construction Management Firm (CMF) due to some technical reasons. (iii) The CEC implementation progress suffered due to initiation of new proposal for engaging private sector and closing of CEC. (iv) Closure of CEC took quite a long time and finally the construction work was transferred to C&W Department and CEC closed in 1998. (v) The restructured targets were achieved by the C&W Department. However, the new initiatives to utilize the unallocated funds taken up at a late stage could not be fully achieved. 7. Restructuring of the Project in 1997 saw a shift of emphasis from the civil works component to quality improvement by reducing the original targets of sites from 3,600 to 2,400. All components of the project were implemented in one formn or the other. As stated above, the Project could not strengthen EMIS appropriately despite providing about 70 computers and other accessories. Similarly, the EDU could not really take off. The ICR rightly observed that the impact of the Project on Project objectives could not be appropriately analyzed. This was so as no formal attempt has been made to carry out complete or sample survey study. 8. Most of the quality components being carried out by the Project have now been taken up by the DoE and are being funded through regular budget of the Department. These include: (i) Shifting of EDU positions from development to non-development. (ii) Sufficient allocation has been made to EMIS to carry out its activities effectively. 29 (iii) Training of teachers and head teachers in the subjects of Mathematics, English and Science (MESH). (iV) Provision of scholarships to girl students for enhancing enrollment and retention. (v) Cash incentive to teachers. (vi) Training of teachers in TA and SRMs. 9. The lessons learned, as depicted in the ICR, are endorsed. These will be helpful to the DoE/GOP and the Bank in designing and implementing new projects in the future. One such lesson is regarding the project period. Initially, the project was slow, it picked up gradually, especially after its restructuring in 1997. The project performance peaked only in the last two/three years of its life. No extension was provided. Had the project period been extended for a year or two, it would have been possible to better capitalize on the project successes/initiatives - leading to a more sustained impact of the quality components. 30 IBRD 23354 aThis p has seer prepared by The WoldBnkd e e-a, f eooee- C - I e rnd 5 eo s.ely for the . n:rrerrel oe of The Wor,d B-nk r a Approximate Line AFGHANISTAN G oop Ihe denominrrVoanused Attock * of Contrd i! AFCHANISTAN 5 i; od the oouda.ies show- - c J_ m 0 ~~~~o,rapoo.,p,rre RawlIpi ndif Oprf o' The WrId Bok Gr-up,: ATTOCK RAWALPINDI a-y o.d emenr the .9 -u fcy ter.s-y JAMMUAJIfZK X / _ X -,. t y en~dorsersednrr or ccepYtafveo ]AkL ID AMMU u , orln pOj b av A U oun oordries , .V ArI D~ AND\ RAWALPINDILSIckO NORTH WEST FRONTIERChakwal .... -', ,YASHMIR f , PAKISTAN i HKA helum _ ' PROVINCE SAR.GOD~~HAMAA _ < - , MIANWALI ]HUM AGUJRAN r 3Mianwali.- Guirat -.f,iaA 7r .' ~~~~~~~~~Khushalb SIJR A ASaLKOT NORTHWEST FRONTIER ,t.,,, j. Scirgodha -Guiranwala "t# - stee°u PROVINCE ~~~~SARGODHA GUJRANWALA n ,,d,i3 SARGODHA ,/ ? ?' ~~~~BHAKKAR',.--o S ,, . r SH~~~~~~~~~EKHUPUPA? * ?? ° ~~Bhakkar ' r ~ . >c /~@a t24s, -sa r JHANG - (3 gMh~~~~~~~~~~Sekhupura L, Lho,e rJhang FAISALABAD BALUCHISTAN D.C n FAISAABAD 2R, | D.-G. KHAN J o ~~~~Singh kk raN AU? ?2 ~ ~ ~ ~ ~ IC .JSahiwoll KK A r D .G . Kh Hn A MULTAN Vihri _ E eR |FAJANPUR - _g .(LM R e3 / _l ~~~~~Bahawalpur ir J / BAHAWALPURe PAKISTAN tcov Rahmyr-h. BAHAWAIPUR .PUNJAB MIDDLE SCHOOLING Rahinnyar-Khan ~PROJ ECT . RAHIMYAR-KHAN . _ ZE ,, ? ~~~~~~~~~~~~~PROVINCIAL_EDUCATION DEPARTMENT: o MAIOR CITIES SINDH T . - DIVISION BOLINDARIFS 3 ECORPORATIN,S , - ~~~~~~~~~~~~~DISTRICT BOUNDARIES * * NATIONAL CAPITAL 3~~~~~ 4u VIHARI r, 3~~~~~~~~~~~~~~ROICEBONARE -~~~~~~~~~~~~~~~~~~~~~~~~ - - . I NTERNATIONAL BOUNDARIES .Lhore (City) nd Lahore (CVihtorenfl are a? rct oF thea somers edmirriefrefive district. However. TRIBAL AREA - romf ah point of view of tIre Prournoica. Education .Daerlrraent, fthey ear two aeparafe districts RAJAN RJANUARY1992