Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) Report Number: ICRR0021309 1. Project Data Project ID Project Name P146451 Sindh GPE Country Practice Area(Lead) Pakistan Education L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-18672,TF-18673 29-Sep-2017 65,730,921.00 Bank Approval Date Closing Date (Actual) 02-Apr-2015 29-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 66,000,000.00 66,000,000.00 Revised Commitment 65,730,921.00 65,730,921.00 Actual 65,730,921.00 65,730,921.00 Prepared by Reviewed by ICR Review Coordinator Group Xiaoxiao Peng Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) was stated in the Grant Agreement (page 7) as well as in the Project Appraisal Document (PAD, page 6) as follows: "The Project Development Objective is to strengthen the institutional capacity to generate, disseminate and use information to support the implementation of key reforms under SESP." Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) Note: SESP refers to the Sindh Education Sector Plan, which had four pillars: strengthened governance and accountability to improve service delivery and achieve desired education outcomes; enhanced equity in education access and participation; improved quality and student learning outcomes; and sustainable sector financing. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Systems strengthening (total costs: US$ 8,208.3 million at appraisal; US$ 3,753.7 million actual. Global Partnership for Education (GPE) contribution: US$ 59 million at appraisal; US$ 59 million actual). As part of a performance-based GPE grant disbursing against the achievement of agreed Disbursement-Linked Indicators (DLIs), this component aimed to support three reform areas that were expected to strengthen the institutional capacity to generate, disseminate, and use information, which was critical for ensuring quality design, implementation, and monitoring of key reforms under SESP. The three reform areas were: (i) Establishment and operationalization of a Sindh School Monitoring System (SSMS). (ii) Establishment and operationalization of a Human Resources Management Information System (HRMIS). (iii) Strengthening of communication activities with internal and external stakeholders to promote the design, implementation, and monitoring of key SESP reforms. Component 2: SESP support (total costs: US$ 26.1 million at appraisal; US$ 18.43 million actual. GPE contribution: US$ 7 million at appraisal; US$ 6.73 million actual). This component was to fund technical, advisory, and capacity-building support (including goods and services) to develop/design, implement, and monitor the activities of Component 1. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: According to the PAD (page 8) and ICR Datasheet (page 2), the total project cost at appraisal was US$ 8,293.4 million. Based on information provided in the ICR (page 9), actual total costs can be calculated as US$ 3,772.13 million. Financing: At the appraisal stage, the project was planned to be financed by both GPE and the Borrower, with 0.8 percent (US$ 66 million) from GPE and 99.2 percent (US$ 8,227.4 million) from the Borrower. The US$ 66 million from GPE was in the form of two Trust Fund grants, one of US$ 44 million (TF018673) and one of $22 million (TF018672). As for the actual amount, GPE contributed US$ 65.73 million (the full US$ 44 million from the first Trust Fund, and US$ 21.73 million from the other), with the small difference from appraised costs caused by exchange rate fluctuations (ICR, page 9). The actual contribution from the Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) Borrower was mistakenly shown as zero in the ICR's datasheet (because the actual amount was not captured by the system) but was confirmed by the project team as US$ 3,694.7 million for the first component and US$ 11.7 million for the second component, for a total Borrower contribution of US$ 3,706.4 million. Borrower Contribution: As mentioned above, the planned Borrower contribution was US$ 8,227.4 million (99.2%) at the appraisal stage. The actual contribution can be calculated from reported data as US$ 3,706.4 million. Dates: The project was restructured two times during implementation.  On September 21, 2015, a level 2 restructuring was approved to correct the respective closing dates of the two GPE trust funds, moving the closing date of one trust fund (TF018672) from September 29, 2017 to June 30, 2016. The rationale was to be consistent with the Minutes of Negotiations for the project, which stated that “the first portion (TF018672) was expected to be used to finance expenditures during the first year of the Project, whereas the second portion (TF018673) is expected to be used to finance expenditures during the second and third year of the Project.”  On September 29, 2017, another level 2 restructuring was approved to extend the project closing date by three months, from September to December 2017, to ensure the completion of technical assistance activities. 3. Relevance of Objectives Rationale The Sindh provincial government is responsible for funding and managing primary and secondary education services. The mandate for education provision is implemented by the Education and Literacy Department (ELD). At appraisal, the education sector in Sindh was weak as measured by primary enrollment ratio, gender parity index, and student learning outcomes. Poor education outcomes were linked to the underperformance of the public school system. To reform the education sector, the government of Sindh launched the Sindh Education Sector Reform Program (SERP) in FY2007/08, which was integrated into the Sindh Education Sector Plan (SESP) in 2014. The World Bank has continuously provided financial support to the SERP and SESP, through the Sindh Education Sector Development Policy Credit (FY07-08, US$ 100 million), the Sindh Education Sector Project (FY09-12, US$ 2,172 million), and Second Sindh Education Sector Project (SEP II, FY13-19, US$ 333 million). The PDOs of the project are highly relevant to the Country Partnership Strategy (CPS, FY2015-19), which was still in effect at closing of the project. The CPS has four strategic pillars: (1) transforming the energy sector; (2) supporting private sector development; (3) reaching out to the underserved, neglected, and poor; and (4) accelerating improvement in services. The project’s PDOs aligned with the third pillar on inclusion and fourth pillar on service delivery by supporting interventions addressing information, accountability, and institutional capacity. It supported the CPS outcome 4.3 (Increased school enrollment and adoption of education quality assessment) and 4.4 (Adoption of performance and transparency mechanisms in selected institutions). The SSMS and HRMIS developed by the project were to provide Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) mechanisms for tracking student enrollment, attendance, and teacher placement and absences. Through developing the SSMS and the HRMIS, strengthening communications with internal and external stakeholders, and other support for capacity building, the project was to contribute to improving the monitoring and management of the public education system in Sindh. The project was also in line with CPS outcome 3.2 (Reduced vulnerability for groups at risk), which included a target on increasing girls’ gross enrollment rates for primary education by 15 percent. The project supported the key reforms under SESP and was complementary to the implementation of SEP II. It directly aligned with the SESP Pillar 1 on governance and accountability, especially three out of the eight strategic results: to develop/establish an independent data collection and monitoring system; to effectively manage human resources (HR); and to increase transparency and involvement of stakeholders in decision-making. The project’s focus on strengthening institutional capacity was also beneficial to activities under SESP Pillar 2 (Enhanced Equity in Education Access and Participation) and Pillar 3 (Improved Quality and Student Learning Outcomes). Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Strengthened institutional capacity to generate information to support the implementation of key reforms under SESP. Rationale To generate useful information for the education sector reform, the project planned to create information systems, including the SSMS for school-level indicators and HRMIS for managing teaching and non-teaching staff. Outputs  Design of the SSMS, including software and manual with data collection formats and procedures, as well as a reports generator.  Hiring and training of Directorate General of Monitoring and Evaluation (DGM&E) staff; establishment of 29 District Monitoring Units; and hiring and training of 29 Chief Monitoring Officers and 349 Field Monitors.  Data collection and entry on school-level information such as school status, attendance of staff and students, and facilities.  DGM&E production of monthly and annual SSMS reports, which also included recommendations for decision and action. Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451)  Development of basic HRMIS (B-HMRIS) software and manual, which enabled the generation of a range of HR-related reports.  Under B-HRMIS, development of a rolling database of 150,000 teaching and non-teaching staff at district and sub-district levels, through consolidation of data from multiple sources (with two-thirds of discrepant files resolved) and collection of biometric data of all staff. Review of this database by the HRMIS team for discrepancies. Outcomes The number of districts for which monitoring data of at least 70% of public schools is collected by DGM&E increased from 0 at baseline to all 29 districts, meeting the target (PDO indicator 1). There was a discrepancy between the PAD and the ICR regarding the exact data type covered by this indicator. The PAD specified it as SSMS data only in Annex 1 (PAD, page 17), but the ICR mentioned both SSMS and HRMIS data when discussing this PDO (ICR, page 12, paras. 25-26), stating “[t]he first PDO outcome was fully achieved particularly through the regular generation of updated SSMS/B-HRMIS information covering all ELD schools and employees” (ICR, page 16, para.44). Achievement of this objective is rated Substantial. The project can be understood as contributing greatly to the result achieved, as neither SSMS nor B-HRMIS existed prior to the implementation of this project. However, two minor shortcomings were:  One planned activity (also a DLI), the preparation of bidding documents for the procurement of an upgraded Comprehensive Human Resource Management Information System (C-HRMIS), was not fully achieved.  Although a DLI for years 2 and 3 was that reports were to be generated from the B-HRMIS system following procedures in the B-HRMIS manual, delays in government approval of the manual meant that the project did not produce a third-party verification report as planned. Rating Substantial OBJECTIVE 2 Objective Strengthened institutional capacity to disseminate information to support the implementation of key reforms under SESP. Rationale The project planned to provide information collected by SSMS and B-HRMIS to the ELD and its Region Reform Oversight Committees (RROCs). Planned activities also involved strengthening the communication of ELD, including the design and implementation of a communication strategy, annual review of sector plan implementation, utilization of a web-based dashboard for internal communication, and a web-based complaints management and redressal system for internal and external stakeholders. All of these activities Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) would plausibly have contributed to strengthened information dissemination to support the implementation of SESP reforms. Outputs  Development of a DGM&E dashboard presenting analytics of SSMS data for the ELD’s internal use.  Employee reports shared with DGM&E to enable SSMS field monitors to have access to up-to-date information on school staff.  B-HRMIS database regularly shared with the districts for corrections.  A range of HR-related information provided to ELD for management purposes.  ELD’s central, regional, and district staff trained on using B-HRMIS.  A communications strategy approved and implemented by ELD, including the establishment of an Information and Communication Unit.  The creation and operation of a web-based internal communications platform to route all department official notifications through its dashboard, and to provide users with other forms of communication (e.g. email, web-based chats and fora, and common-use calendars). The share of "official annual notifications" of the department routed through the internal communication platform increased from 0% to 80%.  The strengthening of a web-based communication system (entitled ILMI), which enabled teachers, parents, community members, and district officials to submit queries, complaints, and requests for responses, resolution, and official processing.  Annual reviews conducted by ELD of the implementation of SESP with internal and external stakeholders.  Dissemination of information related to a range of SESP issues via ELD's website, radio, press releases, social media, and quarterly reports in various formats. Outcomes DGM&E provided monthly SSMS reports to all District Education Offices (DEOs) and quarterly reports to the RROCs. The number of districts for which monitoring data was analyzed and disseminated to ELD management and RROCs increased from 0 at baseline to 29 at the end of project, exceeding the target of 23 (PDO indicator 2). Achievement of this objective is rated Substantial. Minor shortcomings were:  One relevant intermediate results indicator was not tracked during the implementation of the project. The wording of this indicator varied: "positive change in awareness and understanding of SEP II and SESP reforms” in the PAD, and “awareness and understanding of reforms” in the ICR. As planned, the project was to define the unit of measure with an index value, establish the baseline prior to July 2014, and conduct a stakeholder survey every 18 months. As implemented, only one survey was completed by the end of the project, and no index value was created.  There is a lack of full information on the dissemination of B-HRMIS reports. While the ICR mentioned training of staff on all levels on using B-HRMIS, the ICR did not specify whether staff could generate (and/or customize their own) reports directly from the B-HRMIS system or only receive reports distributed by ELD. Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451)  The ICR pointed out some areas with room for improvement, including presentation of information in the SSMS reports, and extending access to the dashboard for SSMS data could be beyond the department itself. Rating Substantial OBJECTIVE 3 Objective Strengthened institutional capacity to use information to support the implementation of key reforms under SESP. Rationale According to the PAD, the data from the SSMS was to inform tangible actions to improve school, sub-district, and district level trends of key school-level indicators, such as student enrolling and teacher presence. The basic HRMIS was to support evidence-based decision making on teacher management. Outputs The ICR mentioned that both SSMS and HRMIS reports were used, as “ELD regularly convoked DEOs and RROC members to review the reports and decide on actions pertaining to absconders, absenteeism, student: teacher ratios, and discrepancies between ELD and Accountant General Controller data," and “at district and regional level for similar purposes, as well as to make decisions on closing schools, allocate civil works, rationalize schools, and resolve boundary disputes pertaining to district authority over schools” (page 13, para. 33). There was no detailed evidence or documentation of these activities in the ICR, however, and no indicators were used to track them over project life. Outcomes The number of districts that submitted B-HRMIS reports to ELD increased from 0 at baseline to 29 at the end of project, exceeding the target (PDO indicator 3). This indicator is problematic, however. The submission of reports does not serve as an appropriate measurement of using information, as it does not measure whether or not any actions were taken consequently based on the reports. In addition, the measurement of this indicator is questionable. The ICR (page 45) noted that the HRMIS reports were produced by the ELD via two reports generators, and ELD staff were trained on how to use them. This information suggests that the ELD was creating the reports for local districts rather than receiving them from local districts. According to one of the DLI indicators, ELD responded to 100% of all complaints received from internal stakeholders within FY 2016/17 through the web-based complaints redressal system within 90 days from the date of receipt, exceeding the target of 50%. The ICR (page 20, footnote 14) stated that this indicator captured only escalated complaints instead of all complaints registered at the first level. The ICR stated that “the vast majority of complaints are addressed immediately at the first level without need for escalation” (page 20), but did not provide evidence on the percentage responded/solved at the first level. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) The ICR listed several improvements in the education sector as a result of the project, including “approximately 6,000 ‘ghost’ or absconding teachers were removed from the ELD’s payroll, a plethora of ‘show cause’ notices were issued, numerous transfers were effectuated between over-staffed and over- crowded schools, and transfer requests were either approved or denied based on ratios in the school from/to which the request was made” (page 13, para. 33). The ICR also mentioned that teacher attendance rates improved from 57% to 61% in districts with SSMS data available, although without providing further information (such as trends of these indicators before, during, and after the project, or comparison between regions with and without SSMS/B-HRMIS), which limits the extent to which these improvements can be attributed to the project. Furthermore, the project is only a part of the efforts made under the SESP, and the SEP II project funded by the World Bank was implemented at the same time, addressing teacher management. Based on the above, achievement of this objective is rated Modest. Rating Modest OVERALL EFF TBL OLD Rationale The first objective was achieved substantially given that the target of the PDO indicator was fully met, although two planned activities experienced delay and were not implemented. The achievement of the second objective is also rated substantial, as the PDO indicator was fully achieved and exceeded, though one intermediate results indicator was not tracked as planned. As for the third objective, the PDO indicator was not constructed well to measure outcome, and the data provided for the related DLI indicator was partial and not adequate. With the achievement of two objectives rated Substantial and one rated Modest, overall efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency The efficiency analysis in the ICR focused on the benefits expected from an increase in primary school completers (who could be expected to earn higher wages) and from the higher productivity that could be expected from improved quality and relevance of education. Improvement in school enrollment—and consequently gains in labor earning—indicate higher efficiency in the education system, and this approach is understood as standard for economic analysis of education system. However, by limiting the efficiency analysis to this standard approach, the ICR largely omitted the project-specific perspective, presenting little evidence on the efficiency with which project’s specific objectives “to strengthen the institutional capacity to generate, disseminate, and use information to support the implementation of key reforms under SESP” were achieved. In other words, the ICR’s efficiency analysis did not focus on how reasonable the project costs were in achieving the project’s objectives, in comparison with project-specific counterfactual scenarios (such as not developing Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) and implementing SSMS and HRMIS, or developing alternative school monitoring and HR management systems). Paragraphs 33 and 38 in the ICR do describe how SSMS and HRMIS reports are being used, and further describes desirable outcomes in terms of reducing teacher absconders and improving teacher attendance rates, but the next step—of analyzing the specific economic benefits of these desirable outcomes in comparison to project costs—was not taken. On a side note related to the economic analysis of improvements in education, an inconsistency noted was that the cost-benefit analysis in the PAD measured benefits derived from an improved secondary school completion rate, while the IRR analysis in the ICR treated benefits as derived from the primary school completion rate and from improvements in the quality and relevance of primary education. On implementation efficiency, the ICR (paragraph 43) notes that the project benefited from having an experienced task team leader (TTL) who was also the TTL of SEPII. The ICR also noted that financial management and procurement were rated moderately satisfactory in implementation status reports (ISRs) throughout the project. However, shortcomings in implementation efficiency were noted. The ICR (paragraph 43) stated that the project experienced delays during the first half of the implementation with issues in procurement management and staff recruitment, though these issues were resolved in the latter half of the project. Also, some planned activities had not taken place by project completion (as noted in paragraphs 34, 37, and 39). Based on the above, Efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  10.00  Not Applicable 0 ICR Estimate  12.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the project objectives is rated High, given that they were firmly aligned with the CPS (FY 2015-19) and SESP. Efficacy is rated Substantial, as two out of three objectives were rated Substantial and one was rated Modest due to the weak construction of outcome indicators and lack of sufficient evidence. Efficiency was rated Modest, as the IRR analysis in ICR confused the efficiency of the project with the efficiency of the education sector. Together, these ratings are consistent with moderate shortcomings in the project’s design and implementation, and consequently Outcome is rated Moderately Satisfactory. Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome As indicated in the ICR (page 23), there are multidimensional risks to the development outcome, ranging from financial and human resources to broader institutional capacity and political economy challenges. The ELD is likely to confront insufficient budgets for operating the HRMIS given Pakistan’s uncertain economic outlook. Economic woes are already seen in the lack of adequate salary expenditures under Eligible Expenditure Programs for the project. The ICR also pointed out the risk of staff turnover and lack of strong leadership for operating systems (SSMS, HRMIS, and ILMI) created by the project. In addition, data collection activities for SSMS could continue to confront challenges in remote areas and areas with risky security conditions, and such challenges are unlikely to be resolved in near future. The development outcome could be further hindered by local resistance. ELD staff at various levels have already experienced strong opposition from entities in the sector who are unwilling to have their performance monitored. 8. Assessment of Bank Performance a. Quality-at-Entry The project objective was closely aligned with both the CPS and education sector strategy. Given the World Bank ‘s continuous and ongoing support to Sindh’s education sector over the past decade, the project was designed to play a critical role in strengthening the quality of public education. According to the PAD, the financing of the project was designed to be results-driven, with the disbursement of majority of the financing (US$44 million out of US$66 million) conditional on the achievement of DLIs. The PAD also clearly identified key risks and mitigation measures related to stakeholders, implementing agency capacity, and political economy concerns. The implementation arrangements were largely based on those under the parallel SEP II. While this approach was logical, since the objectives of these projects were complementary, it did not fully take into account the organizational structure of the ELD. As stated in the ICR (page 22), the Reform Support Unit (RSU) under the ELD was initially assigned to manage HRMIS activity but was unable to take on such responsibility due to legal issues. As a result, the ELD had to reassign the responsibility to a newly created Directorate on Human Resources and Training, which caused delay in implementing the HRMIS. In addition, the PAD provided only a list of results and monitoring indicators (Annex 1 on results framework and monitoring) without explaining the underlying assumptions of and the sequence among them. There was no PDO indicator to measure one of the project's three objectives. The PAD did not address the data source/methodology for some of the indicators, also suggesting weakness in designing Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) the M&E arrangements. In addition, the initial Grant Agreement did not reflect the correct closing dates of the Trust Funds, and the project had to go through a level two restructuring to amend the closing dates. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The Bank team monitored implementation with five supervision missions conducted during the three-year life span of the project and frequent communications with the implementing agency. The same TTL was in place throughout implementation and led the complementary SEP II at the same time. The TTL’s stability and ground knowledge were reported as beneficial to the project. The project experienced delays during the first and second years due to issues with funds transfers, procurement, and staffing. The Bank team identified these issues and worked closely with the implementing agency on resolving them. Implementation progress improved, and the project was completed with a three-month no-cost extension. Concerns about the delays and recommendations were well addressed in ISRs and Aides- Memoire. Shortcomings in financial management and procurement were reflected in MS ratings through all four ISRs. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD outlined the institutional arrangements for M&E, with the ELD and its Reform Support Unit (RSU) and DGM&E responsible for data collection. However, there were some shortcomings in the design of indicators. For objective 3, the PDO indicator did not measure the use of information but rather the dissemination of information; the submission of reports does not serve as an appropriate measurement of using information, as it does not measure whether or not any actions were taken consequently based on the reports. Some intermediate results indicators listed in the PAD had weak linkage to PDOs, e.g., the number of students in public school, and government expenditure in education sector. The PAD did not specify data collection methods for all indicators. b. M&E Implementation The M&E plan was, in general well implemented. The project tracked progress on the three PDO indicators and DLI indicators, and data were collected and analyzed in a methodologically sound Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) manner. The most important moderate shortcoming was that no indicators were added to measure PDO 3. In addition, the ICR's description of production of HMRIS reports suggested that the ELD was creating the reports for local districts rather than receiving them from local districts (see Section 4). There were other minor shortcomings in M&E implementation. Not all intermediate results indicators were tracked as planned. For the indicator on positive change in awareness and understanding of SEP II and SESP reforms, the project planned to establish a baseline index value by July 2014 and conduct stakeholder surveys every 18 months. By project closure, only one baseline survey had been conducted in 2016, with no index value established. Another example is the indicator on project beneficiaries. The PAD defined this indicator as both (1) the number of students in public primary, middle, and elementary schools, and (2) the percentage of students who are female. None of the four ISRs, however, tracked the number of students, and all mistakenly reported the percentage of students who are female as the number of students. This error was identified in the ICR (page 29). There were also minor shortcomings in the measurement of the intermediate results indicators. For example, the indicator on official notifications of ELD/RSU routed through the dashboard did not consider the reach and coverage of the dashboard. The ICR noted that access to the dashboard was only available to DGME. If the indicator only measured provision of official notifications to DGME staff, then its measurement of dissemination of information was limited. The indicator on responses to complaints was similarly limited (see Section 4). The verification for one intermediate results indicator, the implementation of the B-HRMIS in accordance with the HRMIS manual, did not occur as planned. A third-party verification report was mentioned in the PAD, but did not take place due to delay in government approval of the manual and consequent delay in procurement for conducting the validation. For another DLI on HRMIS in FY16-17, which includes “RSU has prepared the bidding document(s) for the procurement of the comprehensive HRMIS,” there is conflicting information in the ICR. The ICR, on one hand, stated that all DLIs were achieved by project closure (page 18), but on the other hand, it recognized the incompletion of this DLI – “the bidding documents for contracting the comprehensive HRMIS have yet to be prepared” (page 13) and “the planned results pertaining to C-HRMIS (included in the DLI matrix) were not fully achieved” (page 15). c. M&E Utilization According to the ICR, project M&E data were discussed regularly and led to course corrections during project implementation. The ELD continues to use the M&E arrangements (SSMS, B-HRMIS, internal notifications systems, and ILMI) developed by the project, an indication of their quality and utility. Application of the M&E mechanism of the project has improved the M&E capacity of the ELD, especially its DGM&E (ICR, page 20). M&E Quality Rating Substantial 10. Other Issues Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) a. Safeguards The project did not trigger any of the World Bank‘s safeguard polices. No indigenous groups were identified, and the project did not involve major physical works with environmental implications. The environment assessment was classified as low risk (Category C). Both environmental and social risk were rated low in all four ISRs. b. Fiduciary Compliance The project experienced minor delays due to procurement issues. At the initial stage, a delay in transfer of funds from the ELD to relevant project delivery units resulted in delays in procurement of staff and consulting firms. During the first half of the project cycle, a long-standing vacancy of a procurement specialist position at the implementing unit (RSU) and the weak capacity of the procurement team had negative impacts on procurement compliance and performance, leading to delays in achieving three out of four DLIs. The project was restructured to get a three-month extension for completing all procurement and financial activities. According to information provided by the project team, Interim Unaudited Financial Reports (IUFRs) were submitted to the Bank within 45 days of the close of each semester, and audited Annual Financial Statements were submitted to the Bank within six months of the close of each fiscal year. IUFRs were reviewed and considered acceptable by the task team. External audit was conducted by office of the Auditor General of Pakistan (AGP), which is acceptable to the Bank as an external auditor. Observations highlighted by AGP in management letters were discussed and resolved in the Departmental Accounts Committee. The ICR (page 21) noted that the project's financial management team was adequately staffed, and that the internal control environment was assessed as adequate throughout the project cycle. Shortcomings in assets management were pointed out by the Bank team and found improved in a mission conducted right after project closure. c. Unintended impacts (Positive or Negative) The ICR (page 17) noted that the web-based complaints redress system enabled participation of a wide range of stakeholders, including students, parents, and community members, strengthening citizen engagement. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) Efficiency is rated Modest primarily due to shortcomings in the economic analysis with Outcome Satisfactory Moderately Satisfactory respect to efficiency of achievement of project-specific objectives. Quality at entry is rated Moderately Satisfactory due to Bank Performance Satisfactory Moderately Satisfactory shortcomings in the results framework and implementation arrangements. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Defining a solid theory of change with valid and measurable indicators is key to assessing a project’s outcome. In this project, it was difficult to measure achievement of PDO 3 without a proper indicator. The data on some indicators were also problematic given inconsistencies in definition and lack of clarity in measurement. Furthermore, the lack of an explicit results chain linking specific inputs and outputs to corresponding outcomes made it challenging to attribute outcomes to inputs. For examples, education sector salary expenditure and non-salary expenditure were regarded as intermediate outcome indicators without a specific explanation of what specific project activities were envisioned to lead to an increase in salary expenditure. Proper institutional arrangements and cooperation among relevant government agencies is critical for enabling and maintaining results. This project’s delay in developing the B-HRMIS was partially caused by reassigning responsibility to a new unit. Due to legal issues that emerged during implementation, the Reform Support Unit was replaced by a newly established Directorate on Human Resources and Training. Cooperation among agencies was sometimes weak, which hindered achievement of results in some cases. The B-HRMIS relies on data regularly provided by the Accountant General Controller, but a reliable system for regular provision of data has not yet been developed. Client commitments and ownership matter greatly in delivering and sustaining results. A lack of stable and committed leadership at the Directorate on Human Resources and Training has imposed risk on continuing operations of systems developed by the project. Moreover, the performance-based mechanism promoted by the project has received resistance from those with vested interests (those who are reluctant to have their performance measured, and those who benefit from resource allocation inefficiencies). Confronting such resistance will require strong leadership from the government as well as a sense of ownership felt by schools and local communities. 13. Assessment Recommended? Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sindh GPE (P146451) No 14. Comments on Quality of ICR The ICR provided a comprehensive review of the project, with discussion aligned to the development objectives. It focused on results and provided some additional evidence beyond the project's formal results framework to demonstrate achievement of outcomes. There were some shortcomings. Most importantly, the efficiency analysis confused project-specific efficiency with efficiency of the education sector. The theory of change outlined in the ICR (page 8) only listed activities, outputs, and outcomes without showing the linkages among them. There were inconsistencies and gaps in reporting of project costs and achievement of DLIs. On fiduciary issues, the ICR did not provide specific information on external audits. The lessons learned in the ICR mostly focused on specific facts, risks, and recommendations, rather than lessons that could be applicable beyond this project. Overall, however, the strengths outweighed the shortcomings. a. Quality of ICR Rating Substantial Page 15 of 15