Document of The World Bank Report No: ICR00001286 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-37750 TF-50854) ON A CREDIT IN THE AMOUNT OF SDR 2.7 MILLION (US$ 3.63 MILLION EQUIVALENT) TO THE Islamic Republic of Pakistan FOR A National Education Assessment System (NEAS) Project December 22, 2009 Human Development Sector South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective June 30, 2009) Currency Unit = Pak Rupees (PKR) US$ 1.00 = 81.3 PKR FISCAL YEAR ABBREVIATIONS AND ACRONYMS ACER Australian Council for Educational Research AEAC Area Education Assessment Centers AIDG Assessment Instrument Development Group AJK Azad Jammu and Kashmir ATC Assessment Training Center B.Ed. Bachelor of Education BISE Board of Intermediate and Secondary Education CBO Community Based Organization CAS Country Assistance Strategy CMU Country Management Unit CW Curriculum Wing DFID Department for International Development DoE Department of Education DRTA Deputy Resident Technical Adviser EFA Education for All EMIS Education Management Information System FANA Federally Administrated Northern Areas FATA Federally Administrated Tribal Areas FCE Federal College of Education (Islamabad) FDE Federal Directorate of Education (Islamabad) GCTE Government College of Teacher Education GoP Government of Pakistan HEC Higher Education Commission of Pakistan IBCC Inter-Board Committee of Chairman IER Institute of Education and Research (University of the Punjab) ICT Islamabad Capital Territory IDA International Development Association IRT Item Response Theory JEA Joint Educational Adviser LARE Lead Assessment Regulatory Expert LMT Lead Master Trainer M.A. Master of Arts M.Ed. Master in Education MoE Ministry of Education MoU Memorandum of Understanding MTR Mid Term Review NEAS National Education Assessment System NEC National Education Census ii NEMIS National Education Management Information System NGO Non-Government Organization NPC National Project Coordinator NPCC National Planning and Coordination Committee NWFP North West Frontier Province PAD Project Appraisal Document PEACE Provincial Education Assessment Centre PIP Project Implementation Plan PITE Provincial Institute of Teacher Education RITE Regional Institute of Teacher Education RTA Resident Technical Advisor SHC Stakeholder Conference SS Subject Specialists TA Technical Assistance ToR Terms of Reference TIMSS Trends in Mathematics and Science Study TWG Technical Working Group UoM University of Melbourne (Australia) WB The World Bank Vice President: Isabel Guerrero Country Director: Yusupha Crookes Sector Manager: Amit Dar Project Team Leader: Sofia Shakil ICR Team Leader: Leopold Sarr iii PAKISTAN National Education Assessment System CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph 1. Project Context, Development Objectives and Design ............................................... 1 2. Key Factors Affecting Implementation and Outcomes............................................... 4 3. Assessment of Outcomes ............................................................................................ 7 4. Assessment of Risk to Development Outcome......................................................... 14 5. Assessment of Bank and Borrower Performance...................................................... 14 6. Lessons Learned........................................................................................................ 16 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners........... 18 Annex 1. Project Costs and Financing .......................................................................... 19 Annex 2. Outputs by Component.................................................................................. 20 Annex 3. Economic and Financial Analysis ................................................................. 21 Annex 4. Bank Lending and Implementation Support/Supervision Processes............. 22 Annex 5. Beneficiary Survey Results ........................................................................... 23 Annex 6. Stakeholder Workshop Report and Results................................................... 24 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 25 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 27 Annex 9. List of Supporting Documents....................................................................... 29 MAP iv A. Basic Information National Education Country: Pakistan Project Name: Assessment System Project ID: P077288 L/C/TF Number(s): IDA-37750,TF-50854 ICR Date: 12/23/2009 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: TAL Borrower: PAKISTAN Original Total XDR 2.7M Disbursed Amount: XDR 1.7M Commitment: Revised Amount: XDR 2.7M Environmental Category: C Implementing Agencies: Ministry of Education Cofinanciers and Other External Partners: UK Department for International Development (DFID) B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 04/17/2002 Effectiveness: 12/19/2003 12/19/2003 Appraisal: 04/19/2002 Restructuring(s): Approval: 06/03/2003 Mid-term Review: 06/30/2006 10/18/2006 Closing: 06/30/2008 06/30/2009 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Substantial Bank Performance: Satisfactory Borrower Performance: Moderately Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Not Applicable Implementing Quality of Supervision: Satisfactory Not Applicable Agency/Agencies: Overall Bank Overall Borrower Satisfactory Moderately Satisfactory Performance: Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 75 75 Sub-national government administration 25 25 Theme Code (as % of total Bank financing) Education for all 100 100 E. Bank Staff Positions At ICR At Approval Vice President: Isabel M. Guerrero Praful C. Patel Country Director: Yusupha B. Crookes John W. Wall Sector Manager: Amit Dar Michelle Riboud Project Team Leader: Sofia Shakil Ameer Hussein Naqvi ICR Team Leader: Leopold Remi Sarr ICR Primary Author: Leopold Remi Sarr F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The development objectives of the National Education Assessment System were to: (i) design and administer assessment mechanisms; (ii) establish administrative infrastructure and capacity for assessment administration, analysis and report writing; and (iii) increase stakeholder knowledge and acceptance of assessment objectives and procedures. In particular, the NEAS was planned as a sample-based national assessment, to be conducted at Grade 4 and Grade 8, in four subjects - Language, Mathematics, Science and Social Studies/Islamiyat/Life Skills. This project aimed to provide the basis for pilot testing the administrative process of assessment, develop the instruments for measuring learning levels and design the information dissemination strategy associated with ii education assessment. In addition, NEAS had this peculiarity that it focused on enabling the Government of Pakistan (GoP), and Provinces/Areas to make systematic assessment a permanent feature of the country#s education system. The key performance indicators of the project include: (i) valid and reliable baseline data for Grades 4 and 8 in four subjects and student achievement over time; (ii) technically proficient assessment staff who are on the regular budget, trained and in place; (iii) partnering arrangements in place; (iv) media campaign, brochures and information bulletins produced, audience appropriate reports of findings produced and disseminated; (v) national and provincial/areas assessment centers' capacity further strengthened in specific areas of sampling, test development, analysis, and report writing. The first indicator does not seem to be clearly defined as, by definition, an indicator is expected to vary over time. Moreover, its associated baseline value and the target value appear ambiguous in the ISRs. For instance, as the project gets implemented, it becomes difficult to continue referring to it as being a non reliable baseline. The last two ISRs pointed to the absence of a reliable baseline whereas the first national assessment # which could reasonably be considered as a valid baseline despite its own limitations as it excludes private schools- was piloted in February/March 2004, at the beginning of the project, and a full scale national assessment conducted in April 2005. Lastly, the progress reported for this performance indicator reflects progress in student performance rather than the establishment of a valid baseline indicator. Overall, it is encouraging to see that the project led to the availability of valid and reliable data for Grades 4 and 8 in key subjects. Revised Project Development Objectives (as approved by original approving authority) (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Valid, reliable baseline data (pilot test scores/measures) for Grade 4 and 8 in four Indicator 1 : subjects and achievement over time. Not possible to fix 2008 Grade 4- values. NEAS is Mathematics: 369; only responsible 2008 Grade 4 for establishing Language Value students learning (Reading): 377; No valid reliable baseline quantitative or achievement and 2008 Grade 4 data available Qualitative) correlates of Language student (Writing): 498; performance 2008 Grade 8 levels, including Science: 477; 2008 teacher input Grade 8 Social iii (provincial Studies: 516 governments are responsible for promoting corrective measures in teacher Date achieved 06/02/2003 06/30/2009 03/31/2009 Comments The implementation of four rounds of assessment has allowed to establish a (incl. % reliable baseline indicator of student performance in Grade 4 and 8. achievement) Technically proficient assessment staff who are on regular budget trained and in Indicator 2 : place. All technically proficient Staff have been assessment staff is appointed under in place on regular regular budget, Staff in provincial centers budget; and to except in Sindh and are appointed with little Value strengthen FANA. Capacity proficiency in student quantitative or capacity in the building is on track, assessment systems. Qualitative) system, staff are with key staff National staff is yet to be retained and participating in appointed. additional resource relevant courses persons from including WBI in- various country training. subdepartments. Date achieved 01/30/2004 06/30/2009 05/26/2009 Comments Sindh staff are in the process of being integrated in the recurrent budget but (incl. % FANA will only regularize its staff by 2012. achievement) Indicator 3 : Partnering arrangements are in place. MOU signed with two institutions At least one (IER Punjab and Pakistani Federal College of Value institution has Education) and they quantitative or No arrangements existed. started admissions are working with Qualitative) in courses on the TA firm and the student NEAS / PEACEs. assessment. IER has completed four courses. Date achieved 12/20/2003 04/30/2008 05/14/2009 Comments (incl. % Satisfactory achievement achievement) Media Campaign, brochures and information bulletins produced, audience Indicator 4 : appropriate reports of findings produced and disseminated. Value All stakeholders Seminars held. No arrangements existed. quantitative or are informed at NEAS and iv Qualitative) every stage of the provincial/areas student assessment centers are and, at least, bi- circulating annually about the quarterly level of student newsletter; website learning, and launched. A draft communication communication strategy under strategy has been implementation. finalized, which now needs to be implemented. Date achieved 02/28/2004 06/30/2009 05/12/2009 Comments (incl. % Partially achieved achievement) National and provincial/areas assessment centers' capacity further strengthened in Indicator 5 : specific areas of sampling, test development, analysis, and report writing. Several international and Additional national trainings required training were held , imparted, but Most key staff have been including WBI in targeting more provided some form of country training Value provincial/area long-term/short-term held in areas of quantitative or staff (and other training, but need further sampling; test Qualitative) resource persons skill development in development; from Education specific areas. analysis, report Departments to writingAdditional build critical training on mass). statistical analysis also conducted. Date achieved 05/02/2008 06/30/2009 05/26/2009 Comments (incl. % At least 75% of NEAS staff training conducted achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Comparative analysis of achievement of grade-4 students in Math and Language. Instruments and Baselines have been No system of monitoring analysis further established for all Value student learning improved, and subjects for Grade 4 (quantitative achievement, so no baseline for and Grade 8 (see or Qualitative) baseline available. private sector also PDO 1 above). established. Comparative v analysis reports for achievement of Grade 4 students in Math and Language have been written. Date achieved 02/28/2004 06/30/2009 10/31/2008 Comments Satisfactory achievement of indicator although the actual value achieved (e.g. (incl. % baselines established) is not related to the intermediate indicator achievement) Indicator 2 : Faculty of the partnering institutions is trained. Faculty of IER Punjab and Fed. All required College of faculty members Education have Value No partnering are trained and received training (quantitative arrangements existed certificate and through workshops or Qualitative) beflore. degree courses on and 8-week student assessment certificate courses. are developed. IER is offering degree courses. Date achieved 02/28/2004 11/30/2007 10/31/2008 Comments (incl. % Fully achieved achievement) Communication strategy developed and implemented for dissemination and use Indicator 3 : of assessment information to improve the system. Deeper sharing of findings and implications with Finalize and pilot primary the Value No communication stakeholders is communication/inf (quantitative strategy system existed required (teacher ormation or Qualitative) before. trainers, textbook dissemination writers, etc. ), and strategy. the communication strategy has been finalized. Date achieved 02/28/2004 03/31/2009 05/26/2009 Comments (incl. % Partially achieved achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 12/01/2003 Satisfactory Satisfactory 0.00 vi 2 05/06/2004 Satisfactory Unsatisfactory 0.00 3 06/28/2004 Satisfactory Satisfactory 0.18 4 12/27/2004 Satisfactory Satisfactory 0.18 5 06/02/2005 Satisfactory Satisfactory 0.42 6 12/08/2005 Satisfactory Satisfactory 0.62 7 06/14/2006 Satisfactory Satisfactory 0.95 8 12/20/2006 Moderately Satisfactory Moderately Satisfactory 0.99 9 06/20/2007 Moderately Satisfactory Satisfactory 1.41 10 12/07/2007 Moderately Satisfactory Satisfactory 1.62 11 06/03/2008 Moderately Satisfactory Satisfactory 2.20 12 12/11/2008 Satisfactory Satisfactory 2.26 13 06/11/2009 Satisfactory Satisfactory 2.64 H. Restructuring (if any) Not Applicable I. Disbursement Profile vii 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal The Government of Pakistan has long recognized the importance of student learning as a yardstick to measure the health of an education system. Consequently, it has made the commitment to not only raise enrolment but also to improve the quality of education throughout the entire system. This commitment has been reflected in the government's policy documents (National Education Policy 1998-2010, Education Sector Reforms 2001 - 2005) and in its reaffirmation at international forums (Jomtien Declaration 1990 and EFA Assessment 2000, Dakar) to include school quality outcomes such as student achievement scores, as performance monitoring indicators in the education sector. In addition, the Curriculum Wing (CW) of the Ministry of Education (MoE) was given the legal mandate to maintain the `standards of education'. In the late 1990s, the Government engaged a policy dialogue with various stakeholders that emphasized the need of a national assessment system to systematically measure and monitor the learning levels of students. As a result, the Government proposed a key initiative of a National Education Assessment System (NEAS) to support the education sector reforms aimed at improving the overall quality of education in Pakistan. The need for a national assessment body was acknowledged when the Federal and Provincial Governments took some assessment initiatives under previous development projects. Most of these assessment activities were confined to the Project level and did not attempt to address systemic assessment issues. In other words, they did not have an institutional base either at the Federal or the Provincial level to sustain assessment activities beyond the Project periods. Consequently, there was little evidence of the impact of Project investments in training and human resource development as far as student assessment is concerned. All the initiatives were on a small scale and lacked systematic standardization and comparability not only across but even within Provinces/Areas. Specifically, it was found that neither the traditional school examinations, nor the external public examinations administered for certification purposes, nor the selection tests for admission to institutions of Higher Education sufficiently fulfill the criteria of standardization and comparability required of a student assessment system against curriculum benchmarks. Although the Ministry of Education (MoE) had testing mechanisms for assessing individual student performance used as the basis for determining promotion and placement, it had no system of measuring how well schools were doing in implementing the curricula and raising student learning levels. Information about student achievement from traditional school examinations, or from the external examinations conducted by the Boards of Intermediate and Secondary Education, was not comparable across the country as the marks were derived from different tests or examinations. Consequently, the Ministry had no institutional mechanisms for assessing the quality of its education service delivery. A dialogue about the need and mechanism for establishing a National Education Assessment System (NEAS) as an integral part of the education system in Pakistan was initiated during a national workshop at Lahore in December 1997. This initiative was followed by a series of National/ Provincial workshops and Technical Group Meetings (TGMs). Consensus about Federal and Provincial roles in conducting assessments, and the broad dimensions of learning assessments, was reached at the TGM held in Islamabad from February 18-19, 2000. A series of Provincial Workshops and National TGMs helped to further build consensus, better define Federal and Provincial/Area roles, and develop Federal and Provincial/Area PC-Is, to equip the NEAS with resources and assure its sustainability. It was also proposed that NEAS and the Provincial Education Assessment Centers (PEACE) would be developed with Federal and Provincial coordination. The Federal unit of NEAS at the Curriculum Wing (CW) in MoE had a coordinating role to support the assessment initiatives in the Province / Areas. PEACEs were established in the Departments of Education (DoEs) to undertake assessment activities in the Provinces/Areas. The preparatory phase of NEAS was completed in the TGMs in December 2001 and January 2002. The Government of Pakistan therefore requested the Bank's assistance in developing NEAS and setting up an institutional framework that systematically measure and monitor student learning, thereby helping assess the quality of the education service delivery in Pakistan. The Bank agreed to provide Technical Assistance along with DFID as this bold initiative was in line with the CAS which stated that the "Bank support will focus on those interventions in the education sector that would help the goal of achieving universal primary education, with particular focus on improving the quality of education provided, and giving priority to the poor and disadvantaged, particularly girls and children in rural areas." 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) The development objectives of the National Education Assessment System were to: (i) design and administer assessment mechanisms; (ii) establish administrative infrastructure and capacity for assessment administration, analysis and report writing; and (iii) increase stakeholder knowledge and acceptance of assessment objectives and procedures. In particular, the NEAS was planned as a sample-based national assessment, to be conducted at Grade 4 and Grade 8, in four subjects - Language, Mathematics, Science and Social Studies/Islamiyat/Life Skills. This Project aimed to provide the basis for pilot testing the administrative process of assessment, develop the instruments for measuring learning levels and design the information dissemination strategy associated with education assessment. In addition, NEAS had this peculiarity that it focused on enabling the Government of Pakistan (GoP), and Provinces/Areas to make systematic assessment a permanent feature of the country's education system. The key performance indicators of the project include: (i) valid and reliable baseline data for Grades 4 and 8 in four subjects and student achievement over time; (ii) technically proficient assessment staff who are on the regular budget, trained and in place; (iii) partnering arrangements in place; (iv) media campaign, brochures and information bulletins produced, audience appropriate reports of findings produced and disseminated; (v) national and provincial/areas assessment centers' capacity further strengthened in specific areas of sampling, test development, analysis, and report writing. The first indicator does not seem to be clearly defined as, by definition, an indicator is expected to vary over time. Moreover, its associated baseline value and the target value appear ambiguous in the ISRs. For instance, as the Project gets implemented, it becomes difficult to continue referring to it as being a non reliable baseline. The last two ISRs pointed to the absence of a reliable baseline whereas the first national assessment ­ which could reasonably be considered as a valid baseline despite its own limitations as it excludes private schools- was piloted in February/March 2004, at the beginning of the Project, and a full scale national assessment conducted in April 2005. Lastly, the progress reported for this performance indicator reflects 2 progress in student performance rather than the establishment of a valid baseline indicator. Overall, it is encouraging to see that the Project led to the availability of valid and reliable data for Grades 4 and 8 in key subjects. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification Neither the PDO nor the key indicators were revised. 1.4 Main Beneficiaries, No specific group is expected to benefit from the Project but the country as a whole will gain from the establishment of a national assessment system. However, the target population for the four national assessments was a carefully selected sample of all students in grades 4 and 8 in government schools and community schools. Teachers and school management or PTA were also key target groups. Finally, the target group also included staff of Ministry of Education (MoE), provincial and area education departments, staff in national, provincial and area assessment centers for their capacity building in all areas of assessment. 1.5 Original Components (as approved) (I) Institutionalization of NEAS The first and the foremost component supported by the Project relates to institutionalizing a National Education Assessment System by establishing an infrastructure to enable systematic assessments. The absence of institutional arrangements at the national level for systematic measurement of student learning had precluded the Government from determining how well the Pakistani education system is performing. As a result, there was no solid empirical basis for making policy decisions about where more resources or additional support was needed or which aspects of service delivery to improve. Similarly, it was difficult to ascertain: (a) how well the curricula were being translated into knowledge and skills among students; (b) the main determinants of student performance; and (c) the ways to assist and foster teachers and schools to use data to improve student performance. (II) Capacity Building in Assessment Along with the administrative strengthening and institutionalization of NEAS, the GoP's objective was to develop the capacity of the Provinces /Areas of Pakistan through training and technical assistance to enable them to conduct sample-based periodic student assessments in line with the Project objectives. Specifically, NEAS was to develop the institutional capacity of MoE (CW) as well as that of Provinces /Areas through their respective PC-1s to monitor the standards of education in the country. NEAS also partnered with national and international institutions to establish a certifying and accreditation body and help institutionalize the training capacity in the country. (III) Stakeholder Knowledge of Assessment A third key component identified by the Project was the need to increase awareness and knowledge about student assessment. Through Stakeholder Conferences (SHC) which disseminate NEAS findings, the Project's goal was to generate informed discussions about the quality of education in Pakistan, what students are learning in school and how to improve the standards of education as well as its relevance to the labor market needs. 3 1.6 Revised Components None of the Project components was revised during the project implementation period. 1.7 Other significant changes The Project was extended for a period of one year, from the original ending date of June 30, 2008 to June 30, 2009, to enable the implementation of a number of capacity building activities, the development of a communication strategy (given the earlier delays in start-up) and to initiate the institutionalization of NEAS. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry Despite being a new Project with a unique design, NEAS has managed to incorporate lessons from previous education projects that emphasized improvement in the quality of both primary and secondary education. Cognizant of the importance of developing a system that assesses and monitor student learning, Pakistan has been committed, since the early stages of the project preparation, to raising the standards of education. Through a participatory approach evidenced by numerous stakeholder conferences, the design phase has allowed the Government, in particular the MoE, to be at center stage in the process of establishing a solid national assessment system. The Project preparation was accompanied by a determination of the risks and mitigation measures. The latter were generally accurately identified except for the political uncertainty and the security risk. 2.2 Implementation The NEAS was conceived out of an intensive consultation process both at the Federal and Provincial levels with key stakeholders. The national workshop organized in Lahore in December 1997 set the stage for developing a national assessment system in Pakistan. This initiative was followed by a series of National / Provincial workshops and Technical Group Meetings (TGMs). Consensus about Federal and Provincial roles in conducting assessments, and the broad dimensions of learning assessments, was reached at the TGM held in Islamabad in February 18-19, 2000. The implementation of NEAS has also benefited from the institutional support of the Curriculum Wing (CW) of MoE which received the legal mandate to maintain `the standards of education' in Pakistan, and later of the Policy & Planning Wing. To provide the administrative base for assessment, the NEAS unit was established in the CW of MoE and endowed with assessment professionals, support staff and facilities to undertake the assessment production, analysis and dissemination, along with the PEACE teams across the country1. NEAS 1 NEAS consists of a central co-ordinating body, plus seven Provincial/Area Education Assessment Centres (PEACE/AEAC) in each Province/Area. The Institute for Educational Research (IER) University of Punjab and the Federal College of Education, Islamabad are the Assessment Training Centres of NEAS. The central NEAS plays a leadership role in planning and co-ordinating different aspects of national assessment, monitoring for quality assurance 4 was subsequently placed under the administrative control of the MOE's Policy & Planning Wing, where it remains till present (and until such time long term institutional options are determined and put in place). The capacity building activities of the project, and in particular, in-country training and overseas training were initially slow due to delays in putting in place the implementation and partnering arrangements. Implementation of the planned training to develop technical capacity of staff in the NEAS network was all the more challenging that there was a seventeen month delay between the project launch and the signing of the Technical Assistance contract. Moreover, the delay in overseas training was manifest when the first cohort of NEAS staff left for Melbourne in February 2007 to only return to the NEAS network in December 2007, therefore leaving them hardly six months to provide any technical input to project activities. For the sustainability of the project, it would be important that the staff from NEAS and the PEACEs who returned with master's degrees from University of Melbourne play a leading role in the development of assessment instruments, data analysis and report writing Further, the weak English language and computer skills of staff in the NEAS network constituted an additional constraint that made report writing an onerous task. Staff turnover had hindered an effective implementation of the project and weakened the potential to create a strong pool of technical expertise. In the perspective of project sustainability, it would be critical that NEAS and PEACE be endowed, on a permanent basis, with a sufficient pool of assessment experts and professional staff that can preserve the institutional memory. In fact, five completed implementation cycles with high rates of staff turnover have demonstrated the need for additional technical and management support to NEAS in order to develop sustainable technical skills in the NEAS network. The human resource constraint is also manifest in the fact that, within NEAS unit, appointments to key posts were not made until August 2003 and the full staffing was not achieved until May 2007. Another weakness of the project implementation relates to the communication and information dissemination strategy. It was deemed relatively weak as there was no permanent counterpart form the Government to carry out project activities. The extent of the communication strategy was limited to stakeholder conferences during the project period, and these were held at the provincial/area and national levels. Deeper sharing and wider dissemination of findings was not undertaken, but this forms part of the strategy going forward in order to make effective use of the information emerging from assessments. In the last year of the project, a communication strategy was developed, which will now be implemented. As a result, the information dissemination was rated by the Project as moderately satisfactory. The successful integration of staff into the regular budgets has had the perverse effect of slowing down the disbursement of IDA fund as many assessment activities conducted by NEAS/PEACE could not be submitted for reimbursement. By the end of 2008, the disbursement ratio was still at a staggering 53 percent, partly as a result of delayed startup in implementation and capacity building activities2. Acceptable audited financial statements for 2008 have been of all NEAS processes and products, in preparing a national report and disseminating assessment findings. The PEACEs/AEACs work collaboratively with NEAS in developing items, administering and scoring instruments, analysing data, and disseminating reports. 2 There had been delay in setting up partnership with a technical firm for capacity building. 5 provided to IDA, and advances to regional offices have been reconciled. However, rectified IUFRs for January ­ March 2009 and subsequent quarters have not been provided yet. It was assured that these would be provided to IDA by end December 2009. Therefore, the Project's financial management has been rated Moderately Satisfactory. Finally, there was uneven institutional development of the assessment centers across the various provinces. For example, the FANA PC-1 was only approved in the last year of the Project while Balochistan could not hold its second stakeholder conference due to the fact that the Provincial government's share of costs was not available. Ownership of the assessment centers and the work of NEAS varied from province to province. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization As the NEAS project deals with measurement of student learning and has measurable indicators to help monitor progress in achieving development objectives, M&E activities appear to be a decisive component for a successful project implementation. There is a growing interest in policy discussion about measuring the quality of education and assessing education service delivery (cf. Vincent Greaney's series of books on national assessment systems, 2008; Jishnu Das et al., 2006). The fact that four rounds of assessment have been conducted provides ample opportunity to measure progress in student learning in Grade 4 and 8 for various subjects (Math, Science, Social Studies, Language). The rich database gathered has further helped undertake comparative analysis of Grade 4 students in Math and Language. Such reports are then likely to inform policy decision making as well as help improve data collection and test designs. While, overall, it is encouraging to see that the Project led to the availability of valid and reliable data for Grades 4 and 8 in key subjects, and that analysis of this data is taking place, the challenge is to strengthen utilization of this data and information to help improve the system. There is now a good base for stimulating the use of information on learning gaps by various stakeholder departments to bring improvements in teaching and learning environment. Monitoring of the assessment system was the responsibility of the Ministry of Education's Policy and Planning Wing, where NEAS was housed. However, the provincial systems were housed within the Education Departments' various structures (Bureaus of Curriculum, or as separate centers), and maintained links with the federal NEAS as well as their respective provincial education departments. Overall coordination on policy issues was the responsibility of the National Planning and Coordination Committee (NPCC), which also was responsible for overseeing the vision and strategy development of NEAS. But, although the main structure for monitoring the overall vision for assessment system under the project is the NPCC, a credible monitoring system tracking progress in key school quality indicators is needed to strengthen capacity for research and in-depth analysis of those indicators. The first monitoring indicator of the project does not seem to be clearly defined as its associated baseline value and the target value appear ambiguous in the ISRs. For instance, the last two ISRs pointed to the absence of a reliable baseline whereas the first national assessment was piloted in February/March 2004, at the beginning of the project, and a full scale national assessment conducted in April 2005. This first national assessment could reasonably be considered to be a valid baseline, assuming that the sampled public schools are nationally representative, despite its own limitations as it excludes private schools. Lastly, the progress reported for this performance indicator reflects progress in student performance rather than the establishment of a valid baseline indicator. But overall, it is encouraging to see that the project led to the availability of valid and reliable data for Grades 4 and 8 in key subjects. 6 2.4 Safeguard and Fiduciary Compliance The Government complied with safeguard policies and fiduciary responsibilities during the Project implementation. Ratings were satisfactory in many categories but the financial management reporting was relatively weak. 2.5 Post-completion Operation/Next Phase The project ended in June 30, 2009 and the Government has requested the World Bank and DFID to provide continued support for a second phase of capacity support. World Bank and DFID are in advanced stages of developing a trust fund supported capacity building program. The MoE recognizes that the process of ensuring a sustained and strengthened assessment system will require continued technical assistance and has requested this support for further strengthening and institutionalization of NEAS. Financing for this support would be provided through a recipient executed trust fund facility with funding from DFID and CIDA. There would also be a small Bank-executed component to support key technical assistance activities and other tasks on which the Bank has a comparative advantage. The MoE has therefore agreed to prepare a new 5 year plan, in its vision for NEAS, for capacity development, technical assistance, institutional strengthening, and stronger communication/dissemination strategy. This new plan will build upon the existing one, by revising it to reflect key priorities for the future TA program. This MoE's vision for NEAS embeds Pakistan's participation in TIMSS as it has the potential to effectively prepare the country to this major international event. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation The Bank's assistance strategy is focused on helping reduce poverty in Pakistan by investing in people and promoting sustainable economic growth. Supporting pro-poor policies and promoting gender equity are among the CAS program priorities. In this regard, the CAS reaffirms that "Bank support will focus on those interventions in the education sector that would help the goal of achieving universal primary education, with particular focus on improving the quality of education provided, and giving priority to the poor and disadvantaged, particularly girls and children in rural areas." Among the principal indicators of the quality of education services delivered are what students are learning and how well they are able to apply the skills acquired in the labor market and in real life situation. In line with the CAS's objectives, the NEAS project was designed to measure student learning and to develop an assessment system that would monitor the standards of education in Pakistan. The Project's Development Objectives (PDOs) remained relevant throughout the course of the project, and remain relevant to Pakistan's national and provincial education strategies. While the PDOs were slightly sharpened to give a clearer focus and relevance to the needs of the system, as indicated below by the words in italics: (1) To design and administer a system of monitoring student learning achievement; (2) To establish administrative infrastructure and capacity for assessment administration, analysis and report writing; and 7 (3) To increase stakeholders' knowledge and acceptance of assessment objectives and procedures and to provide feedback to help improve the quality of teaching and learning. An important dimension of the relevance of NEAS objectives has to do with the choice of Pakistan to participate in the famous international assessments known as Trends in Mathematics and Science Study (TIMSS) in 20153. In preparation for this, it is critically relevant that MoE builds the administrative infrastructure and develop the technical capacity of the NEAS network so that Pakistan's education system is well positioned to compete with other countries of similar level of economic development that are taking part in the TIMSS. 3.2 Achievement of Project Development Objectives The focus of NEAS project on developing a national assessment system over the last five years of implementation has resulted in substantial achievements relating to the process of institutionalization of NEAS, the building of an administrative assessment infrastructure, including the promotion of partnerships with national and international institutions and the increase in stakeholder knowledge about assessment. These achievements are reflected in the results presented below: (I) Design and Administer a System of Monitoring Student Learning Achievement Between February 2005 and June 2008, four rounds of large scale student assessments on a sample basis for Grade 4 and 8 and one round of equating studies for two subjects in Grade 4 have been successfully completed and shared with a wide range of stakeholders (for further details, cf. Annex 6b on the outline of NEAS subject assessment framework). The institutional framework for systematically measuring student learning is gradually building up in the country. There is now an adequate pool of trained professionals available in country, both at the national and provincial levels to develop assessment tools and participate in the administration of assessments. Subject specialists and test development specialists have received extensive trainings thereby creating an in-country expertise to conduct trainings on development of assessment tools. As a result, NEAS and PEACE staff have been extensively involved in the development, administration and analysis of assessments. For instance, no international test item was used, only national curriculum items were developed but the level of difficulty of test items was raised to international standards. Anchor items were also used from the 2nd round of assessment; the quality of items used and the grading were improved over time and the tests were administered in Urdu and Sindhi. NEAS has also successfully translated the new curriculum introduced in 2007 by MoE into competencies for measuring learning as future assessments will be based on the new curriculum. Punjab is, for instance, taking the lead in establishing a Provincial Examination Commission that will use competency based assessment in examination system. In the same vein, Punjab is in the process of designing its own district-based testing, which when completed, will allow comparison of school performance across districts, thereby improving the quality of the school decision making. 3 This is the current plan of the MOE, but the final decision about the timing of participation in TIMSS is yet to be taken. 8 (II) Establish Administrative Infrastructure and Capacity for Assessment, Administration, Analysis and Report Writing A good administrative infrastructure had been established at the National and Provincial levels from a zero baseline. National and Provincial management committees have been established to provide advice and support decision making in NEAS. Professional staff in National and Provincial offices has been regularized in five Provinces and areas and it is expected that Sindh will also bring all key assessment staff under the recurrent budget in the coming months leaving only FANA Area to complete it by 2012. All operating costs related to conducting assessments are now financed under the governments' regular budget4. Although staffs are now equipped to implement assessments and conduct analysis, substantial support is needed both at the national and provincial levels to strengthen staff capacity to design, implement and analyze assessments. It is also crucial that the linkages between the National and the Provincial/Area systems are reinforced. Finally, in the medium to long term, the Government needs to pay attention to strengthening the management and leadership of NEAS, and develop a roadmap for moving towards institutional autonomy with a solid administrative and financial apparatus. The Project extension from June 2008 to June 2009 was essential to begin key activities relating to the institutionalization of NEAS. A detailed review was conducted of the options available for institutionalization, including options for establishing NEAS as a credible and autonomous institution, and a Task Force was set up for this purpose. A medium term capacity building program and implementation plan has also been developed. A vision document presenting the Government of Pakistan's long term vision for assessments and quality of education has also been drafted. The vision document sets forth suggestion for the institutionalization of NEAS and a more devolved administrative structure. There are also plans to develop linkages with Higher Education Institutes in Pakistan to actively engage the research faculty in further exploring NEAS findings using rigorous statistical methods. For this in-country partnership with Higher Education institutions, NEAS should give special attention to the relevance of available courses/training for NEAS activities. During the project period, partnering relationships were established with one international and two national institutions but only one partner institution could successfully participate in the program, viz. the Institute of Education Research (IER) at Punjab University. Nonetheless, this partnership has substantially contributed to the development of national capacity to offer specialized courses in assessments. During the Project period, 40 people completed an M.Ed; 33 received short-term training of 3-4 months, 168 persons were offered WBI courses conducted in Pakistan and 1680 were trained through various workshops conducting in the country. The baseline of in-country technical capacity was evaluated as marginal at the start of the project, and substantial progress has been made through these various trainings to build a pool of assessment experts within the education system from which the NEAS network can draw for its own assessment activities. 4 The government has provided Rs. 0.8 million for assessment activities in its 2009-10 budget , which is sufficient to conduct a national workshop. 9 Two national institutions (IER Punjab and Federal College of Education) worked with the TA firm as well as the NEAS/PEACEs. Further, IER Punjab has already completed four courses. NEAS should also explore a partnership with the High Education Commission (HEC) to introduce highly technical specializations such as statistical sampling, test and background questionnaire development, equating, item response theory (IRT) and multivariate analysis at University Departments of Statistics, Psychology, Education and Economics. (III) Stakeholder Knowledge and Acceptance of Assessment Disseminating assessment findings was an integral part of the NEAS Project. The four rounds of assessment conducted have significantly contributed to raising awareness and increasing stakeholder knowledge about assessments. Of paramount significance is the greater acceptance of NEAS as a credible institution, particularly among high level policy makers. NEAS has received wide recognition from the government and donors as an institution that provides comprehensive information on the quality of education service delivery. However, much more work is required to improve the analysis of assessment and to provide feedback into the education system. Based on the national assessment's findings, NEAS has prepared, with the support of education experts, a set of recommendations for education policy makers and implementers. These recommendations have been widely shared with all relevant education stakeholders for review and implementation. However, much deeper dissemination is required so that the feedbacks received can feed into various institutional decisions relating to, for instance, textbook, curriculum or teacher training. To further improve the dissemination of findings, NEAS has prepared a comprehensive communication strategy to reach out to all stakeholders with clear results in order to maximize impact on public policy and planning. Some aspects of the communication strategy such as, up- gradation of the NEAS website, printing of selected communication products, started before the completion of the project on June 30, 2009. Finally, it is worth mentioning that the outputs by component described in Annex 2 are inherently linked to the outcomes achieved during the project. 3.3 Efficiency With a modest investment of about US $5.94 million (including DFID and GoP's shares), the NEAS project has achieved the following: Capacity building component · 15 person months of in-country short-term training (workshops, seminars, etc.); · 40 person months of in-country certificate / degree training; · Around 200 person weeks for participation in international symposia, conferences and study tours; · 10 persons (20 person years) for overseas degree (M.A.) training); and · 10 persons (20 person years) for "sandwich" degree training. Technical assistance component · 36 person months of Long-term TA; · 10 person months (40 wks.) of ST international TA 10 · Around 250 person weeks of short-term national TA; and · 48 months of "on-line" support from partnering institution(s). During the project period, close to 50,000 students of class IV and VIII and their teachers were administered tests in Language (Urdu/ Sindhi), Mathematics, general Science and Social Studies along with background questionnaires for students, parents, head teachers, and teachers were administered. However, the full impact of the project on enrollment in primary and secondary school, on improved quality of schooling in Pakistan as well as internal efficiency is yet to be documented. Furthermore, there are externalities associated with the NEAS project as some countries in the region are now walking in the footsteps of Pakistan to develop their own assessment system (e.g. India, Bangladesh to name few). Teachers and students are also encouraged to raise the standard of learning and performance across Provinces/Areas. Although it is difficult to quantify the economic or financial returns of the project, its impact is nonetheless substantial, especially in regards to the small investment. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory The transfer of key positions5 at NEAS and PEACEs to the revenue budget constitutes a clear performance indicator. The long term objective of MoE is to integrate national assessment within the mainstream education system in Pakistan, as it was the case for curriculum development and teacher training. Towards this end, the staff positions in the NEAS and five of the seven PEACEs/AEACs had been transferred to the government's recurrent budget within the project period. The Sindh PEACE is expecting to get approvals in the coming months while FANA AEAC will only be able to convert staff into the recurrent budget by 2012. The project has therefore successfully managed to build ownership of assessments not only at the Federal level but also across Provinces/Areas. In addition to the financing of these positions under the revenue budget, other recurrent activities (including costs to administer the assessment papers) were also included in the regular government budgets. While only bare minimum operating expenses have been included in the regular budgets, it is nonetheless a good start and indicates government ownership in a short time of a new system. A yardstick by which the project performance can be gauged is the amount of interest national assessment of student learning has generated in the country over the past 10 years, at all levels. Prior to this Project, there was hardly any debate about measuring student learning and very little understanding of the difference between examination and student assessment. Pakistan remains the only country in South Asia that has developed a national assessment system. Further, it has already completed four rounds of national assessment, attesting to the vitality of its system. By June 2008, NEAS had already established baselines for all four subjects of Grade 4 and two subjects of Grade 8. Table 1 provides a summary of the pilot and actual tests administered during the Project period. Annex 6a provides a detailed description of how to conduct a cycle of student assessment. Table 1: National Assessment Tests Conducted over 2004/2008 5 These core positions include: (i) subject specialists (SS); (ii) system analyst; (iii) NEAS and PEACE coordinators; (iv) computer programmer; (v) assessment experts. 11 Year Grade and Subject Test Type 2004 Grade 4 (Mathematics, Languages) Pilot Testing 2005 Grade 4 (Mathematics, Languages) Large Scale Testing Grade 4 (Social Studies and Science) Pilot Testing 2006 Grade 4 (Mathematics, Languages, Social Large Scale Testing Studies, General Science) Grade 8 (Mathematics, Languages) Pilot Testing 2007 Grade 8 (Mathematics, Languages) Large Scale Testing Grade 8 (Social Studies, Science) Pilot Testing 2008 Grade 8 (Mathematics, Languages) Large Scale Testing Grade 8 (Social Studies, Science) Large Scale Testing The technical knowledge of NEAS and PEACE staff has increased over the course of the Project. As compared with the base year of 2003, when there was hardly any understanding of assessment and its process, the NEAS and PEACEs staff were able to understand and engage on technical aspects of the assessment system in a relatively short period of time. All the basic work to establish a system had been completed, thereby providing to the staff a real learning opportunity. Consequently, at the time of Mid-Term Review (MTR), the Implementation Progress (IP) was rated moderately satisfactory. Another benefit of the Project is that the assessments have provided credible evidence of how the education system is performing. They have also provided ways of measuring how well schools are doing in implementing the curricula. The national assessments are based on a nationally-representative sample and provide some comparability across different provinces and areas. This gives to education policy makers, local officials, teachers, schools and other stakeholders, valuable information about where the gaps in learning are and how specific institutions at the provincial and area level (such as teacher training, textbooks, etc.) can be improved. . While these achievements are remarkable, it is worth noting that the high staff turnover at NEAS, the delayed implementation schedule of national assessments and the ambitious NEAS training plan developed at the project design stage, have somewhat reduced NEAS's effectiveness in achieving all its targets. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development The emphasis put on improving the overall quality of education in Pakistan is likely to engender the need for schools, teachers and students to strive for excellence thereby contributing to raising the learning levels. As a result of improving the quality of schooling, the quantity of schooling would inevitably rise, inducing parents, especially those from poor families, to send or keep their children in school. The NEAS, by providing continued credible evidence of student learning, will help to restore the public confidence in the education system. As a result, enrollment would likely increase. The priority given to girls' education in the CAS has translated into a focus given by the Project to increasing female enrolment through improving the quality of learning. Table 2 reveals that, if 12 more girls are now attending school, they are not learning less than boys. To the contrary, girls appear to be performing better than boys in Sciences and Social Studies in grade 8. The Project has also adequately addressed rural-urban and boy-girl disparities through test items, sampling and training in assessment tasks. Lastly, it has further identified inequality in enrollment in rural areas especially for girls. Table 2: Mean Score in Sciences and Social Studies (Grade 8) by Gender in 2008 National Assessment Gender Mean score in Sciences Mean score in Social Studies Boys 473 514 Girls 484 520 (b) Institutional Change/Strengthening The integration of NEAS/PEACE staff into the recurrent budget constitutes a major step toward the institutionalization of education assessment. In addition, the implementation of the fourth round of assessment with little support from international consultants reflects to some extent an institutional shift toward a progressive ownership of assessment activities by the GoP. The NEAS network has developed a close collaboration between the federal Ministry of Education and the Provincial and Area Departments of Education (DOEs). The overall coordination is the responsibility of the Federal government under the umbrella of the broader Education System Reform initiative led by the Federal Secretary of Education. The various working groups responsible for the design and implementation of the assessment system are listed below: Federal government ­ MOE: Overall design and administration; consolidation of test items to be used in administering tests; leadership in pre-administration information campaign; national level reporting and policy making; coordination of technical assistance and training; Partnering /Implementing institution(s) (Pakistan and International): item development, technical assistance, training, coordination of pilot testing and analysis; setting standards for administration and sampling, help in conducting assessment; Provincial/Area governments: item development for pooling at the national level, provide list of schools and other data for sampling at national level; help in collecting background data of sample schools/students etc.; test administration, consolidation, analysis and dissemination of provincial findings. (c) Other Unintended Outcomes and Impacts (positive or negative) The fact that Pakistan is succeeding in establishing a credible national assessment system may have inspired other developing countries, especially those in the region, to follow the pioneering path paved by Pakistan. It is believed that, by strengthening the nascent NEAS institution, Pakistan could rise and meet the challenge posed by the next round of TIMSS international test. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops None. 13 4. Assessment of Risk to Development Outcome Rating: Substantial A number of risks had threatened the development outcomes identified by the NEAS Project: (i) high staff turnover in the NEAS network; (ii) political instability exacerbated by threats to national security has precluded international consultants and firms from providing continued support to the Project and has limited Bank supervision; (iii) risk that assessment experts would lose their technical expertise given the quest for professional growth as they climb the administrative ladder. To mitigate these risks, a number of measures had been identified: (i) intensive, high quality and continued dialogue between the Bank team and government; (ii) strong emphasis on continued capacity building for the full institutionalization of NEAS and the mainstreaming of assessment into the education system; and (iii) ability to use video-conference supervision in the event of restricted mission. Despite these mitigation measures, the huge uncertainties justify the substantial rating of the risk to development outcomes. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance The Bank overall performance rests on the performance achieved in ensuring quality at entry and quality of supervision. (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory The Bank performance for quality at entry was rated as satisfactory. The Bank provided the TA in response to a need expressed by the GoP to develop a national assessment framework that would help systematically measure student learning levels and the overall quality of education service delivery. The government's request was aligned with the Bank's CAS which put a special emphasis on improving the quality of schooling. The Bank team remained extensively engaged in policy dialogue with the Federal Ministry of Education as well as the Provincial Departments of Education since the beginning of the Project. The stakeholder conferences held in early 2000 provided the Bank another opportunity to further explore the idea of national assessment. Notwithstanding this satisfaction, some areas could have been improved during the Project preparation: (i) some performance indicators such as reliable baseline or the comparative analysis of achievement were not clearly stipulated and it was unclear how their progress or target value was measured; (ii) the design could have also incorporated the substantial risk associated with the uncertainties of national security to potentially include mitigation measures regarding the hiring of international consultants or firm for building technical capacity of the NEAS network. (b) Quality of Supervision Rating: Satisfactory The Bank performance for ensuring the quality of supervision was rated as satisfactory. The Bank's task team was composed of Human Development, Finance and Private Sector, Financial Management staff and international and national consultants who contributed significantly through country and sector knowledge. The international expertise was much needed as Pakistan was the first country in the region to embark on building a national assessment system. Moreover, some of the team members have had long standing engagement with the Federal and Provincial 14 Governments, in particular MOE and the provincial Department of Education, which has established the credibility of the Bank in policy dialogue. This continuity in the Bank team has helped maintain a coherent dialogue with Government counterparts during supervision. It is noteworthy underlining that the collaboration between the Bank and DFID ­ co-financier of the NEAS Project who also provided support to the Government in other areas- has helped avoid duplication among Development Partners in their effort to assist the GoP. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory Overall, the Bank performance has been satisfactory. During the five year implementation, the consistent dialogue with senior Federal MoE officials and Provincial government counterparts as well as other assessment stakeholders, the international expertise brought in and the key role played by the CMU in supporting the Project, have led to acknowledge NEAS as a credible education assessment structure not only within Pakistan but even outside the country. Today, the policy discussion gyrates around ways to strengthen the existing national assessment system rather than how to build an assessment system. Continuity in the Bank team has sustained a permanent dialogue with high level policy makers and has contributed to improving the quality of supervision missions which included experts from various sectors. Finally, the aide memoires summarizing progress in project implementation and recommendations for better project monitoring have been relatively well written and documented. Many of them have suggested ways to smooth the project implementation process. 5.2 Borrower Performance The Government and the implementing agency are the same. Therefore, the rating and justification are given below in section (c): (a) Government Performance Rating: (b) Implementing Agency or Agencies Performance Rating: (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory The Borrower's performance is rated as moderately satisfactory for the following reasons: (i) Although the country has successfully established a credible national assessment system, the project management has not performed to the expected level. For instance, there was a seventeen-month delay between the Project start and the signing of the TA contract. In addition, the development of a holistic communication strategy was not undertaken till late into the Project period. Further, there were weaknesses in financial management reporting: (i) shortcomings in the Jan-Mar'09 IUFRs have not been addressed yet; (ii) IUFRs for the subsequent quarters have not been provided to the Bank; (iii) audited financial statements were generally received with 15 some delay and (iv) Auditors' opinion has been a mixed bag of unqualified and qualified exception. (ii) Throughout the life of the Project, disbursement has been lagging behind the appraisal estimates. Only a sum of SDR 1.74 Million (64.5 percent) could be disbursed verses signed amount of SDR 3.63 million. This was mainly because of the slow implementation and the fact that many activities against which expenditures incurred were already financed under the regular government budget, the Implementing Agency was reluctant to seek further claim on reimbursements (despite repeated advise from the Task Team to encourage the Project to seek reimbursement).. In addition a sum of GBP 0.677 million (60.13 percent) was also disbursed under TF No. 50854 verses signed amount of GBP 1.127 million. (iii) The rapid staff turnover and the delay in appointments to vacant posts in NEAS meant that consistent capacity building over a period of time through in-country and overseas training could not be achieved to fully support the Project activities. The delay in setting up partnership with a technical firm for capacity building has also resulted in slow implementation of key training activities of the Project. (iv) NEAS dissemination conferences have unquestionably generated awareness about education assessment. However, there is urgent need to go beyond stakeholder conferences to develop dissemination materials targeted to specific audiences and policy planners in order to maximize the use and impact of findings. Further, NEAS and the PEACEs/AEACs held two stakeholder conferences in 2006 and 2007 whereas Balochistan was only able to hold one6. 6. Lessons Learned The NEAS Project has contributed not only to generating acceptance of assessment system among Pakistani but also to increasing knowledge about student learning throughout the country. While this achievement is commendable and unique in the region, much work is required to raise student learning levels to international standards: (i) With strong ownership of assessment by the Government through the integration of staff into the revenue budget, the true institutionalization will require moving towards a semi- or autonomous body so that the credibility of NEAS is fully established technically, financially and administratively. (ii) Since the four rounds of assessment have been successfully conducted, Pakistan could validly consider collecting panel data and developing the national expertise to rigorously analyze them, thereby contributing to the overall improvement of education service delivery. More specifically, NEAS should consider a longitudinal assessment design which allows reporting gain scores and progress in performance over time. This would provide MoE with a unique opportunity to introduce interventions such as teacher incentives, provision of teaching materials and textbooks, vouchers for school choice, and for the relevant research units to investigate their impact over time. 6 Note however that a national SHC was organized in 2008. 16 (iii) The discrepancy between the planned samples and the actually achieved samples in the four rounds of assessment points to the need to improve the administration of the tests to allow a more representative sample of schools and students. In this regard, NEAS should ensure that, during the regular round of national assessment, the sampled public schools are all interviewed so that they can have a valid representation of Government schools in Pakistan. (iv) Furthermore, to establish a solid baseline at the national level, there is an urgent need to incorporate in NEAS sampling frame the growing number of private schools mushrooming across the country as they are yet to be included. This is not a miniscule task but a complete electronic sampling frame for private schools which became available from NEMIS in 2007 is likely to facilitate the choice of sampled schools. However, the complexity of the sampling frame (with multiple categories of schools by ownership, declared and actual medium of instruction etc.) indicated that only a limited pilot study on a sample of Urdu medium private schools could be conducted by NEAS in 2008. Much work remains to be done if private schools are to be included in NEAS assessments to draw valid inferences about student achievement from representative samples, and to compare performances across types of schools (government versus private schools). In the 2008 assessment, NEAS has also piloted its assessments in a sample of Urdu medium private schools, drawn from the list of private schools, now available in the NEMIS database from the National Education Census (NEC). (v) The NEAS item pool is not yet rich enough to support the development of tests with comprehensive coverage of the national curriculum at all grade levels. Given the limited number of items in its pool, NEAS is limited to developing parallel tests instead of tests using the Balanced Incomplete Block (BIB) design recommended by consultants. The BIB design would allow for wider curriculum coverage with multiple test forms. Given that the NEAS sample is large enough, it could support such expansion of the item pool. NEAS could also support the inclusion of national assessment related topics in the Assessment and Evaluation papers in B.Ed. and M.Ed. courses of Punjab University and possibly of all universities in Pakistan that provide B.Ed. and M.Ed training, through the HEC. (vi) As one of the main objectives of NEAS is to identify correlates of student achievement, future dissemination activities of the NEAS network should present findings from analyzing the rich assessment database and produce targeted reports for various stakeholder groups to get feedbacks that could, in turn, feed into the assessment system. (vii) In light of the weaknesses in financial management observed during the Project cycle, Accounts Officers should not be involved in administrative work as this leads to weak controls and also delay in submission of financial reports. Furthermore, since there was only one Designated Account established under the IDA Credit, the fund flow to the provinces was problematic. To overcome the fund flow problem, each implementation agency should have its separate Designated Account. (viii) Lastly, as the NEAS network grows and assessment staff become part of the Government administration, there is need to foster technical leadership within NEAS and to reinforce vertical linkages between NEAS and PEACEs, cross-fertilization within PEACEs and horizontal linkages between Partnering Institution (IER, Punjab University) and the NEAS network so that Pakistan can valuably compete with other countries in international tests. 17 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies The Borrower has prepared its own completion report whose main points are summarized in Annex 4. (b) Cofinanciers DFID has prepared its own completion report which is summarized in Annex 8. (c) Other partners and stakeholders None. 18 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) CAPACITY BUILDING 5.09 3.83 75.2 PILOT TESTING AND IMPLEMENTATION OF 1.39 0.37 26.6 ASSESSMENT ACTIVITIES INFORMATION 0.12 0.19 158.3 DISSEMINATION Total Baseline Cost 6.60 4.39 66.5 Physical Contingencies 0.13 0.00 0.00 Price Contingencies 0.20 0.00 0.00 Total Project Costs 6.93 4.39 63.3 Front-end fee PPF 0.00 0.00 .00 Front-end fee IBRD 0.00 0.00 .00 Total Financing Required 3.63 2.64 72.7 (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Borrower 2.16 0.55 25.4 UK: British Department for 1.99 1.2 60.0 International Development (DFID) International Development 3.63 2.64 72.7 Association (IDA) 19 Annex 2. Outputs by Component Component/Activity FY 2004/05 FY 2005/06 FY 2006/07 FY 2007/08 FY 2008/09 1. Design and Administer a System of Monitoring Student Learning Achievement National Assessment Round Pilot 11977 students 11954 students 14144 students 11493 students 784 schools 788 schools 763 schools 708 schools Grade 4 - 2 subjects Grade 4 - 4 subjects Grade 8 - two; Grade 4 - two Grade 8 - 2 subjects 2. Establish Administrative Infrastructure and Capacity for Assessment, Administration, Analysis and Report Writing 2.1. NEAS Network Paid by the Government NEAS staff Project staff Project staff in Revenue budget PEACEs/AEACs staff Project staff Project staff Revenue budget in Sindh Balochistan, FATA, Punjab, Punjab, AJK, NWFP 2.2 Technical Assistance From 2004-2009 Total international TA input 46 person months Total national TA input 62 person months Online support from Partner Instit. 48 months 2.3 Capacity Building From 2004-2009 In-country short-term training 15 person months In-country certificate / degree training 40 person months Overseas degree (M.A.) training 20 person years "Sandwich" degree training 20 person years 3. Stakeholder Knowledge and Acceptance of Assessment National Stakeholder Conference 1st held 2nd held 3rd held Report of NEAS findings Witten Witten Witten Witten Disseminated Disseminated Disseminated Disseminated 20 Annex 3. Economic and Financial Analysis At Project appraisal, no economic and financial analysis was carried out. Section 3.3 provides a short note on the efficiency of the TA investment. 21 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Supervision/ICR Asif Ali Senior Procurement Specialist SARPS Umbreen Arif Education Spec. SASED Anwar Ali Bhatti Financial Analyst SACPK Vincent M. Greaney Consultant WBIGC Tahseen Sayed Khan Operations Adviser SACBD Amna W. Mir Program Assistant SASHD Hasan Masood Mirza Consultant SARPS Ameer Hussein Naqvi Consultant SASHD Hasan Saqib Sr Financial Management Specia SARFM Huma Ali Waheed Research Analyst SASED (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY02 14 59.55 FY03 25 38.63 FY04 0.04 FY05 0.00 FY06 0.00 FY07 0.00 FY08 0.00 Total: 39 98.22 Supervision/ICR FY02 0.00 FY03 0.00 FY04 21 23.10 FY05 24 29.94 FY06 4 13.21 FY07 13 67.67 FY08 27 80.15 FY09 20 0.00 Total: 109 214.07 22 Annex 5. Beneficiary Survey Results (if any) 23 Annex 6. Stakeholder Workshop Report and Results (if any) 24 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The borrower has produced a project implementation report covering the various key aspects and issues relating to the Project. The main points are summarized below: Project Implementation Status: The Project was fully implemented with four rounds of national assessment and four pilot tests along with background questionnaires for students, parents, head teachers, and teachers administered in languages (Urdu, Sindhi), Mathematics, Social Studies and Science for grade 4 and grade 8. Technical capacity building of staff from NEAS, provincial and area assessment centers, education departments and NEAS partnering institutions was done through in-country, international, short term and long term training. Assessment findings were disseminated and recommendations for actionable interventions by Federal Provincial / Area and District authorities were disseminated through three national and 10 provincial stakeholder conferences. All key education stakeholders at Federal, Provincial/Area and school level participated in NEAS training, test administration, monitoring and dissemination activities. Acceptance of assessment objectives and procedures has increased as demonstrated by Sindh PEACE, Punjab PEACE and UNICEF building upon the findings of NEAS. Through quality TA provided in the Project the NEAS is now equipped with high level technical skill sufficient to carry out all assessment functions. Valid instruments and procedures are now in place to link future survey data with the baseline data established through the assessments. Implementation and Operational Experience: Lack of continuity and ownership by one wing of the Ministry of Education led to delays in approval and thus in activities. Just the approval of the PC I took one year. Project activities also suffered because of lack of funds and / or delays in their release on part of the government. This was further aggravated due to a high turnover of staff and slow recruitment process. No recruitment rules were in place initially and those that were later added were inadequate for the requirement of subject specialists. Main reasons contributing to the high turnover were both lack of a career path offered to staff and absence of a retention plan for those on deputation. As the Project depended heavily on trained staff and there was a scarcity of it, this was a serious problem for the completion of activities. Also there was no provision for backup staff to continue assessment tasks when some staff members were away on training. The Project had a shaky start when the training of staff was setback due to a delay of 14 months in approval of the TA firm. Once in place maximum benefit could not be derived from the TA firm due to communication problems, lack of coordination in terms of timing and cultural mismatches for relevance to the Pakistani context. On the other hand local partnering institutes like IER, Punjab University and FCE lacked the capacity to live up to the requirements of the Project. Translation and back translation of local languages to English was not envisaged at the beginning of the Project and this turned out to be a major issue during the activities. The NEMIS data used as the basis of the study had a lag of one year and this caused implementation issues in field. Finally coordination between the NEAS and provincial area centers was difficult and this led to slow progress of certain activities. The need for appropriate staffing provision for dissemination of assessment findings and mechanisms for intervention was not foreseen. Despite issues and a very aggressive project plan all targets were achieved and assessment is now recognized as a system for identifying weaknesses and improving the quality of education. Sindh and Punjab have already begun their own tailored assessment based on the learning from NEAS and Pakistan is now making efforts to participate in TIMSS in 2015. 25 Some of the operational issues still being faced by NEAS include lack of appropriate space and infrastructure. NEAS needs to be institutionalized and recognized for it to continue to benefit the quality of education in Pakistan. 26 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders Indicators Progress and Comments Valid, reliable baseline data Not possible to fix values. NEAS is only (pilot test scores/measures) for responsible for promoting corrective measures in Grade 4 and 8 in four subjects; areas such as teacher training. Improvement in Achievement overtime for trends expected over time and inclusion of private grade 4 students in four subject sector. Technically proficient All technically proficient assessment staff is in assessment staff who are on place on regular budget; and to strengthen capacity regular budget trained and in in the system, staff is retained and additional place resource persons from various sub-departments. Partnering arrangements are in At least one Pakistani institution has started place admissions in courses on student assessment. Pilot tests administered on Completed on schedule schedule Regular meetings o f National All stakeholders are informed at every stage of the Planning and Coordination student assessment and, at least, bi-annually about Committee are held; the level of student learning, and communication strong and weak areas of strategy under implementation student learning identified; policy makers informed to develop strategies and action plans for improving curriculum, teaching, and resource allocation; and equipment purchased Information marketing Strategy developed and in process of finalization campaign activated; and information products finalized. Workshops, seminars held for stakeholders · Education assessment system is new to Pakistan. The federal government is committed to the education assessment. The government has regularized the NEAS and its ongoing activities are now funded from the government recurrent budget. The government has also requested the donors to provide support for second phase of the program. Long term support will help achieve the purpose that will contribute to Goal. · Assessment mechanisms and instruments have been designed and administered: This has been successfully accomplished, national staff has been extensively involved in the development, administration and analysis of assessments, NEAS was able to conduct the last round of assessments with minimum support from international consultants. There is now adequate pool of trained professionals available in country at national and provincial level to develop assessment tools and participate in assessment administration. Subject specialists and test development specialists have received extensive trainings and there is in-country expertise available to conduct trainings on development of assessment tools. Between February 2005 and June 2008, four rounds of large scale student assessments on a sample basis for Grade 4 and 8 and one round of equating studies for two subjects in 27 grade 4 have been successfully completed and shared with a wide group of stakeholders. NEAS has also successfully translated the new curriculum introduced in 2007 by Ministry of Education into the competencies for measuring learning as future assessments will be based on the new curriculum. · The provincial PEAS performance is uneven. Better in Punjab but other provinces specially NWFP and Baluchistan are slow in institutionalizing the system · NEAS team is in regular contact with the provincial PEAS to embed the assessment system. · World Bank is the lead donor in this Project and their financial and technical contribution has been substantial. World Bank national and international technical resources were utilized in developing assessment methodology, developing items, conducting data and analyzing it. · The learning achievement base line was developed for grade 4 and grade 8 and in subsequent years NEAS and PEAS were able to generate follow up reports. Now NEAS is able to design and conduct assessment on the basis of its own human resources. · Most of the priority reforms have been met, but the institutionalization of NEAS required and further capacity building of staff. A Task Force has been notified by the Ministry of Education to work on the institutionalization of NEAS. 28 Annex 9. List of Supporting Documents Project Documents 1. Project Appraisal Document, The World Bank, 2003 2. Implementation Status and Results Reports, The World Bank, 2003-2008. 3. Aide Memoires, The World Bank, 2003-2008. Other Documents and Publications 4. NEAS reports on Assessment findings, 2005-2008 5. Stakeholder Conference reports, 2005-2008. 6. Draft ICR by Government of Pakistan, 2009. 7. British Council Project Completion report (NEAS), 2009. 8. Das, Jishnu, P.Pandey and T. Zajonc, Leaning Levels and Gaps in Pakistan, The World Bank Working Paper, WPS 4067, Dec. 2006. 9. Vincent Greaney, T. Kellaghan, 5 volumes on National Assessment of Education Achievement, The World Bank, 2008. 29 30