Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) Report Number: ICRR0021636 1. Project Data Project ID Project Name P128208 SL-Public Sector Pay & Performance(FY12) Country Practice Area(Lead) Sierra Leone Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51220 31-Oct-2015 13,970,761.52 Bank Approval Date Closing Date (Actual) 31-May-2012 30-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 17,000,000.00 0.00 Revised Commitment 17,000,000.00 0.00 Actual 13,970,761.52 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hjalte S. A. Sederlof Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement (page 4), the Project Development Objective (PDO) of the Public Sector Pay and Performance Project (PSPP) was to improve competitiveness in pay, performance management and accountability of, and increase staffing of middle and senior staff in, the Civil Service in the territory of the Recipient. The Project Appraisal Report (PAD) has the same PDO (page 10 of the PAD). Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) Achievement of the PDO will be assessed against four sub-objectives: (i) improvements in the competitiveness of civil service pay; (ii) improvements in performance management of the civil service; (iii) improvements in the accountability of the civil service; and (iv) increased staffing of middle- and senior levels in the civil service. b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components There were two components: Component 1: support to the government’s public sector reform program (cost estimate at appraisal US$15 million; actual cost US$11.97 million). The component supported the three reform areas specified in the PDO: pay reform, recruitment and staffing, and performance management and accountability. Disbursements on the component were to be made against the achievement of selected “milestones” (disbursement-linked indicators, DLIs) attached to each reform area. Each DLI was part of a step-by-step sequential reform path to achieve the objectives of its respective reform area and the PDO. The project initially had 15 DLIs at US$1 million each, with the option of disbursing 50 percent in advance against each DLI. Reform area 1 - pay reform. In order to attract and retain key professional and managerial staff, a more equitable and competitive pay structure for the civil service was to be introduced. It was to be based on a remuneration survey for establishing salary ranges for each grade, and supplements to be paid for specific jobs. The survey would also offer a perspective on the evolution of demand for skills in the civil service over the longer term. Three DLIs supported progress in this area: completion of the survey; all civil service jobs evaluated and assigned to an approved pay structure; and all civil servants in Grade 6 and above (middle and high level civil servants, representing some 12 percent of the civil service) paid in accordance with the approved pay structure. Reform area 2 – recruitment and staffing. The objective was to right-size and right-skill the civil service. To this end, the government’s Human Resources Management Office (HRMO) was to develop open, competitive, and competency-based recruitment procedures, as well as explicit criteria for recruitment into managerial, professional, and technical grades. Processes were to be established to identify and develop current civil servants who had the potential to take on greater responsibilities in the future. The processes were to be accompanied by a development scheme for high potential junior and mid-level professionals, as well as new promotion procedures. Six DLIs supported progress in this area: recruitment procedures designed and approved; seven staff recruited and trained to manage the recruitment and selection process; at least 60 percent of priority vacancies filled in Project year 1 according to approved recruitment procedures; at least 80 percent in Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) Project year 2; and at least 90 percent in Project year 3; and donor-recruited local technical assistants (LTA) integrated into the civil service. Reform area 3 – performance management and accountability. In order to improve performance in the civil service, increase citizens’ trust in government, and strengthen accountability of ministries to the executive and to citizens. To this end, guidelines for performance contract management were to be developed and applied to annual output-based performance targets and work plans; regular performance reports were to be generated; and corrective action taken to bring actual performance closer to targeted performance. Individual performance appraisals were to be piloted and subsequently rolled out in all ministries. Human resource and performance management units in all ministries were to be fully staffed and trained to facilitate the introduction of performance appraisals. The HRMO was to have completed the design, guidelines, and database on the functioning of the new processes in each ministry, and provided them with feedback. Managers were to be trained in setting individual targets and undertaking effective appraisal discussions. Six DLIs were to support progress in this area: guidelines approved for performance contract management for civil servants at the senior level (Grade 11 and above); one annual cycle of the performance appraisal process for civil servants at the middle level (Grade 7 and above) completed in pilot ministries; at least 80 percent of annual performance appraisal reports for CY 2014 for civil servants in Grade 11 and above, respectively, in pilot ministries are of appropriate quality; and all ministries’ performance against performance targets evaluated, and results published. Component 2 – technical assistance (estimate at appraisal US$2.0 million; actual cost US$2.0 million). The component was to finance a communications program, set up a website for disclosing reform- related information, conduct job evaluations and labor market surveys, and provide technical advisory services on project implementation. Three adjustments were made in March, 2014, to the DLI time frame and the content of some DLIs due to implementation delays: the evaluation of all ministries' performance against performance targets, and the annual performance appraisal reports for civil servants in grades 7 to 10, were dropped, and the retention of recruits to priority vacancies was added. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost and financing. At appraisal, the total cost of the project was estimated at US$17.0 million to be financed through an IDA Credit of US$17.0 million equivalent. Actual project cost at completion was US$13.97 million. For Component 1, U$15 million had been allocated (US$1 million for each of the 15 DLIs). At closure, US$11.97 million had been disbursed for Component 1 - US$10 million against 10 completed DLIs. In addition, advances of US$1.5 million against incomplete DLIs, and US$0.5 million against one of the two dropped DLIs, were disbursed. (As noted in Section 2d above, each DLI was valued at US$1.0 million, with an option to disburse 50 percent of that value as an advance.) The ICR (page 13) states that tracking of the advances for the three unachieved and one dropped DLI was inadequate; subsequent adjustments were not made (see Section 8b below for further details). The allocated US$2.0 million for Component 2 was fully disbursed. Borrower contribution. No Borrower contribution was planned or materialized. Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) Dates. The project was approved on May 31, 2012. It became effective on October 23, 2012. The original Closing Date of October 31, 2015 was extended three times: (I) to October 31, 2016 due to significant delays in achieving the Year 2 targets; (ii) to October 31, 2017, since many key activities were proceeding at a slower pace due to an Ebola epidemic; and (iii) to June 30,2018, to provide more time to complete the project in the aftermath of the Ebola crisis. The total implementation period was 73 months, nearly 80 percent longer than the originally planned 41 months. 3. Relevance of Objectives Rationale The project objectives remain relevant to the country situation, government policy and World Bank strategy. Coming out of a decade-long civil war and facing serious development challenges, there was a need to rebuild a civil service badly weakened during the conflict, and with severe gaps especially among middle level professional and technical staff. To address the issue, the government drew up a medium term public sector reform program covering the years 2011-2015 and focusing on pay reforms, skills gaps, institutional capacity, and performance management. The World Bank’s Country Assistance Strategy Progress Report for Sierra Leone (FY10-13), still valid at project closure, emphasized the need to build capacity for improved governance and service delivery within government. While it did not explicitly address public sector reform, it recognized the need for such reform – it was a cross-cutting issue that concerned the whole administration, and it was likely to be critical as government resources increased with rising iron ore exports. It was also important for improving performance across the Bank’s portfolio: according to the ICR, task teams were signaling capacity problems in public services as a key constraint in delivering operations. As a consequence, the Bank operation was designed to support the last three years of the government's public sector reform program. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective improved competitiveness of civil service pay (pay reform) Rationale Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) The objective was to be achieved by pay reform that would ensure civil servants are paid more equitably and competitively. Pay reform was to include a reevaluation of civil service jobs, development of a new pay structure, and application of the new pay structure to civil servants. Planned remuneration and job evaluation surveys were completed, albeit with delays in the hiring of consultants for the job evaluation survey. However, translation of survey results into a new pay structure has been held up for fiscal reasons. The necessary fiscal space, which was expected to be generated by rising iron ore exports, has not yet materialized. The IMF and other development partners have also been calling for greater control of the wage bill, reducing room to maneuver on pay reform. Consequently, this objective has not been met. Rating Negligible OBJECTIVE 2 Objective Improved performance management in the civil service Rationale The objective was to be achieved by the introduction of performance appraisal and contract management processes focused on mid- and higher level civil servants; by setting performance targets for all ministries; and by introducing a performance appraisal and performance contract management process for civil service managers; setting performance targets in all ministries; and evaluating staff performance with that of relevant non-state actors. The following rules-based performance management tools at the ministerial and individual levels were introduced:  Guidelines for performance appraisal of civil servants in grades 7 to 10, and for contract management for civil servants at grade 11 and above;  Supervisors in pilot ministries at central, regional and district level were trained in performance appraisal skills; and  All pilot ministries have agreed annual performance targets for individual civil servants in grade 7 and above. Some elements of a performance management system have been introduced. However, there is no evidence provided that they have contributed to improved human resource management or of tangible enhancements in civil service performance or productivity. Rating Modest Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) OBJECTIVE 3 Objective Improved accountability in the civil service Rationale The objective was to be achieved by publishing ministries’ performance targets and their achievements, as well as comparing performance to that of relevant non-state actors. However, the primary activities under this objective – evaluation of ministry staff performance and comparing the results with the performance of comparable non-state actors - was dropped during the May 2014 restructuring, as was the DLI on publication of such performance targets. Rating Negligible OBJECTIVE 4 Objective Increased staffing of middle and senior levels in the civil service Rationale The objective was to be achieved by developing appropriate recruitment procedures, filling Grade 6+ vacancies in accordance with annual recruitment plans, and integrating local technical assistants (LTA) into the civil service system. Open, competitive, merit-based recruitment procedures were put into place, drawing on a human resource plan based on the medium-term expenditure framework, and on annual recruitment plans for the period 2013- 2015. Seven staff were recruited to the Public Services Commission to manage recruitment and selection, meeting the outcome target. LTAs were integrated into the civil service system. Although the vacancies were filled in accordance with procedures that had been put into place under the project to enable competitive recruitment, there is no direct evidence regarding the qualifications and quality of the recruits. Rating Substantial OVERALL EFF TBL OLD Rationale Efficacy of the first and third objectives is rated negligible - a more competitive pay structure could not be introduced due to fiscal constraints, while the key activities that were to lead to improved civil service accountability were dropped. The second objective is rated modest - performance appraisal and contract management processes were put in place, but there is no evidence of tangible changes in performance or Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) productivity. Efficacy of the fourth objective - increased staffing of middle and senior levels - achieved its targets and is rated substantial. Overall Efficacy Rating Primary reason Modest Low achievement 5. Efficiency Neither an ex ante, nor an ex post, quantitative analysis of project costs and benefits was undertaken. There were considerable operational and administrative inefficiencies. First, US$2 million was disbursed as 50 percent advances against DLIs that were not achieved. The advances were not recovered. While advances of up to 50 percent were allowed by the Financing Agreement, the latter stipulates that withdrawals would be conditional on satisfactory evidence that DLIs are achieved in order to be deemed eligible expenditures. The advances were not systematically tracked to prevent ineligible expenditures from occurring. Second, there were serious delays, amounting to nearly 80 percent of the original time allowed for implementation. These were partly due to political and macroeconomic factors, and to the consequences of the Ebola epidemic. However, they also reflected design deficiencies. There were fifteen DLIs divided into four sets, each set linked sequentially - there would be no disbursements against a DLI until the indicators for the previous one had been met. Confronting this encouraged the advances already alluded to. Despite the advances and the delays, 12 percent of the IDA credit remained undisbursed at closure. Third, efficiency was negatively affected by a lack of strategic leadership on the part of the Authorities. Leadership Team meetings that were to be chaired by the Minister of Finance occurred infrequently, thereby impeding adaptability to changing circumstances. Numerous implementing agencies reported that sufficient funds to enable them to engage in project activities were not made available (ICR, page 30) Efficiency Rating Negligible a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) 6. Outcome The project's objectives addressed a significant development challenge that was recognized both by the government and the Bank, and their relevance is rated high. Efficacy is rated modest: Achievements under the first and third sub-objectives - a more competitive pay structure and improved civil service accountability - were negligible. Under the second sub-objective - enhanced performance management - procedures were put in place, but there is little tangible evidence as yet that they have led to better performance management; efficacy is, therefore, rated modest Under the fourth sub-objective - increased staffing of middle and senior levels - the targets were achieved, and efficacy is rated substantial. Efficiency is rated negligible: there were grave operational deficiencies, including US$2 million of unrecovered advances for DLIs that were not achieved. Overall, shortcomings are considered major, and outcome is assessed as unsatisfactory. a. Outcome Rating Unsatisfactory 7. Risk to Development Outcome Based on the mostly incomplete record of achievement of important elements of the reform targets under the project, it does appear that there still is significant risk to reaching satisfactory outcomes in key areas – compliance and quality, the recruitment process, and pay reform. Here, the absence of strategic leadership of the reform process probably represents the greatest risk. The team believes that the achievements, such as elements of a performance management system, and recruitment procedures, are unlikely to be reversed. 8. Assessment of Bank Performance a. Quality-at-Entry The project was strategically relevant in that it recognized the need to build capacity in the civil service to better manage expected growth in public resources. The results framework embodied a reasonably robust causal chain linking activities to expected outcomes. M&E design developed key indicators that were measurable and relevant to outcomes. There were, however, a number of major weaknesses in Quality at Entry. First, preparation and design did not sufficiently factor in the need for training and guidance in the implementation of DLI-based disbursements and related advance payments. This lending modality had only recently been introduced to World Bank operations in the Africa Region, and for Sierra Leone, an institutionally challenged country, it was completely new. The PAD and other documents contained inadequate practical guidelines, while provision for training, both for Bank and client staff, was insufficient. The PAD assumed that there would be frequent and intense supervision, but this did not materialize (see Section 8b below). Second, the results-based financing model would also contribute to implementation delays, as many activities proceeded at a slower pace than anticipated, in turn holding back others that could have gone ahead in the absence of sequencing. It did not, therefore, fulfill expectations that it would reduce the risk of slow Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) and uneven progress in a weak institutional environment. Third, risks related to the uncertain political climate and macroeconomic situation were not fully factored in or mitigated. They were to have a serious negative impact on project achievements. Fourth, the three year implementation period was chosen to fit the government’s reform timetable rather than the country's institutional and capacity constraints. It proved to be far too short. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision While the Ebola epidemic caused significant disruptions, there were also major shortcomings in the Quality of Supervision. It was seriously under-resourced. Not only did the more than usually intensive effort foreseen in the PAD not materialize, but resources were only marginally increased (by seven percent) to cope with almost three additional years of implementation. This had a number of consequences. First, advance disbursements against DLIs were not systematically tracked, so that ineligible expenditures (expenditures that should not have been made under the DLI system) arose from non-achievement of some indicators (see Section 5 above). Second, the supervision team could have been more proactive in discussing with the government the need for a level one restructuring, after key assumptions made at appraisal (for example, concerning the fiscal space needed for the establishment of a competitive pay scale) proved to be no longer valid. Third, although changes to the results framework were captured in the Project Paper and Amendments to the Financing Agreement following the March 2014 level two restructuring, they were not formalized into the Bank’s monitoring system. All subsequent Implementation Status and Results Reports (ISRs), therefore, continued to use the initial results framework, thereby further complicating the systematic tracking of the 50 percent advances against achievements of the DLIs. Fourth, again as part of the March 2014 restructuring, the number of DLIs was reduced from 15 to 13, but without any corresponding adjustments to the “value” of each DLI. Fifth, while the supervision team drew attention in ISRs to implementation challenges, there is little indication of action taken to address them. Quality of Supervision Rating Unsatisfactory Overall Bank Performance Rating Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design A detailed results framework and monitoring plan were developed during preparation. Six outcome indicators were to be tracked, two covering each of the reform areas. All indicators were relevant to their Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) purpose and included targets; no baselines were calculated, as the project was introducing new processes. Responsibility for overall monitoring of the results framework lay with the project implementation agency, the Public Sector Reform Unit (PSRU). The PSRU was to coordinate the reporting of four key implementing agencies, each one responsible for a reform area. b. M&E Implementation The PSRU produced progress reports of varying timeliness and quality (ICR (page 33). However, there were significant weaknesses in this process: The PSRU's capacity was weak; monitoring by the government's leadership team, intended to be quarterly, was irregular and infrequent; and the results framework and monitoring plan were not updated following disruptions to the timetable. As noted in Section 8b above, the results framework was not updated following the March 2014 restructuring, and Bank supervision reports continued to rely on the original one. According to the ICR, the PSRU did not use the results framework, and their quarterly reports did not include updates on its status. Throughout implementation, the Unit focused on the attainment of DLI targets. Nonetheless, DLI achievements against advances were not properly tracked. The project's gender target (30 percent of direct project beneficiaries should be women) was not monitored, and at project closure, no gender breakdown was available (it was subsequently estimated by the ICR team). c. M&E Utilization Neither project management nor government decision-makers appear to have made use of the M&E system (ICR, page 34). M&E Quality Rating Negligible 10. Other Issues a. Safeguards No social or environmental safeguard policies were triggered by the project, which was classified as a Category C project for purposes of environmental assessment. b. Fiduciary Compliance Financial management. Portfolio and Risk Management (PRIMA) assessments during preparation and implementation consistently rated the project’s financial management as satisfactory. Still, financial management does raise some questions: Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208)  the ICR notes that the Bank did not track progress on advances to DLIs; still, advances were made against four DLIs that were either never achieved or cancelled during the first restructuring.  the annual external audit only covered expenditures undertaken within the TA Component of the project.  The Audit Service of Sierra Leone reported on the project, but it did not make a distinction between disbursements recorded as advances against non-achieved DLIs and disbursements recorded as reimbursements on achieved and verified DLIs. Procurement. A procurement assessment during appraisal found that the PSRU had inadequate experience and capacity to conduct procurement activities under the project. The hiring of a part-time procurement specialist mitigated this risk, but a May 2017 Procurement Risk Assessment and Management System (PRAMS) rated project procurement Moderately Unsatisfactory. Procurement delays also contributed to a delay in activities, such as the hiring of the consultant for the job evaluation survey. c. Unintended impacts (Positive or Negative) None noted. d. Other Gender. Based on analysis undertaken by the ICR team, the gender target was not met. Female beneficiaries of the recruitment process averaged 18 percent. Only in the case of human resources officers did female recruitment exceed the target, reaching 32 percent. In fact, there appears to have been a deterioration - an analysis undertaken by the Public Service Commission , noted that, in 2015, a quarter of appointees were women. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Efficacy was modest, with two Moderately out of four objectives rated Outcome Unsatisfactory Unsatisfactory negligible. Efficiency is also rated negligible. There were major shortcomings Moderately Bank Performance Unsatisfactory in both Quality at Entry and Unsatisfactory Quality of Supervision. There were major weaknesses in M&E implementation. The Quality of M&E Modest Negligible results framework was not utilized. Quality of ICR --- Substantial Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) 12. Lessons The first three lessons are taken from the ICR, with some adaptation of language, and the fourth is drawn by IEG:  Alertness to incentives can influence collaboration in results-based lending. Teams need to assess the incentive effects of individual DLIs on agency collaboration. In the Sierra Leone case, the client noted that the pay-off was a public good and achieving DLIs was unlikely to result in any proportionate budget increase for the agencies concerned.  Changing country circumstances can affect project outcomes. Teams need to assess the economic and political circumstances that may disrupt project design and require adjustments, for instance during mid-term reviews. In Sierra Leone, the project could have benefited from an early realization of the implications on civil service reform of changing fiscal space.  While focusing on DLIs, the results framework is also relevant. An exclusive focus on DLI sequencing when managing a DLI-based project may not sufficiently define the reform path. In Sierra Leone, a greater focus on the results framework and intermediate indicators might have better brought forth issues on the feasibility of certain reforms. It might have better focused the team's attention to changes in the results framework as DLIs were being adjusted (during the 2014 revisions).  Introduction of new lending modalities, with which Bank staff are also unfamiliar, is best avoided in institutionally challenging contexts. If, nonetheless, they are utilized, then project outcomes require provision for adequate training on both the Bank and Borrower sides, and by enhanced resource availability for supervision. In this case, the use of the DLI model was not accompanied by sufficient training, and the allocations made for supervision would have been barely adequate even without the DLI model, given the difficulties of the operating environment. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a thorough assessment of the project. It covers most sections in sufficient detail, providing enough evidence-based information and analysis to allow the ICRR to be completed. It draws some good lessons, but in many instances they raise issues that don’t appear to be tackled in the text. This is notably the case in reviewing the supervision process and (absence of) revisions to the results framework. Likewise, an explanation of the external auditing arrangements would have been useful. The document is internally consistent, but at 41 pages, it is too long. Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review SL-Public Sector Pay & Performance(FY12) (P128208) a. Quality of ICR Rating Substantial Page 13 of 13