ICRR 13873 Report Number : ICRR13873 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/23/2012 Country : Liberia Project ID : P109195 Appraisal Actual Project Name : Emergency Senior Project Costs (US$M): US$M ): 7.65 Executive Service Project (ESESP) L/C Number : Loan/ US$M): Loan /Credit (US$M): 2.3 2.3 Sector Board : Public Sector Cofinancing (US$M): US$M ): 5.35 5.35 Governance Cofinanciers : UNDP, USAID, Board Approval Date : 10/12/2007 Humanity United, Closing Date : 12/31/2010 06/30/2011 Greece Sector (s): Central government administration (100%) Theme (s): Administrative and civil service reform (100% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stefano Migliorisi Chad Leechor Navin Girishankar IEGPS2 2. Project Objectives and Components: a. Objectives: The project development objective (PDO) was presented in a slightly different way in the Trust Fund for Liberia Grant Agreement (GA), and in the Emergency Project Paper (EPP). On page 8, the EPP stated that the project had a short term and longer term capacity building objective . The short term objective was to deploy “a core group of skilled professionals and managers immediately below ministers that is capable of developing policy and helping the department to implement it .� The longer term objective of the Senior Executive Service (SES) was to help “the ministries and departments they are placed in undergo restructuring and reform, consistent with the Civil Service Reform Strategy .� A key requirement will be for the key ministries to be restructured in the coming three years, with revised mandates, new staffing structures with surplus positions eliminated, job descriptions, and a new pay and grading system, so that the SES, which will start as an enclave system, can be mainstreamed into the rest of the civil service .� The PDO is phrased differently later in the EPP (p.10 and Annex 2). According to the Grant Agreement (p. 5), “the objective of the Project is to assist the Recipient to recruit experienced and qualified professionals with the requisite technical and managerial skills for strategic decision making for improved public service delivery .� Here, the goal of “improved public service delivery � a higher level objective to be achieved through several tools including, but not limited to this project . This ICR review uses the first version of the PDO provided on page 8 of the EPP since it is more precise about the what the project aims to achieve, and it distinguishes between short term and longer term objectives . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The short and long-term objectives of the project were to be achieved by filling an immediate high level capacity gap and by building institutional capacity through recruitment of 100 Senior Executives over three years . The project had two original components : Component A : Recruitment of members for the Senior Executive Service (SES) SES ) scheme (planned and US$ 6.02 million ). This component involved the recruitment of members for the SES scheme . actual cost : US$6 Funding was used to pay for the wages of members of the SES for three years . It was planned to recruit a minimum of 100 SES members. Component B : Management of the SES scheme (planned and actual cost : US$1 ). This US$ 1.63 million ).. component was used to procure basic furniture and office equipment to support and strengthen the capacity of Civil Service Agency (CSA) as the oversight civil service institution to eventually manage the SES program at the end of the project. In addition, the component was used to pay for travel and relocation expenses, workshops and seminars to be organized on the SES scheme, exit package, and monitoring and evaluation o f the scheme. Again, these costs were supposed to be shared among donors, as determined through the Project Implementation Committee (PIC). Amongst the items the Bank will finance is an independent impact assessment of the scheme. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Cost Financial project costs were US$ 7,648,000, US$6,018,000 for Component 1 (salaries of members for the SES scheme) and US$1,630,000 for administration of the SES scheme . Component 2 included inter alia US$ 300,000 for travel and relocation costs, US$ 180,000 for exit packages for civil servants currently occupying the positions and not redeployed, US$200,000 for workshops and seminars, US$ 205,000 for Project Management Unit (PMU) staff salaries and training, US$ 200,000 for assessment of SES candidates by an human resources (HR) firm ad US$70,000 for an impact evaluation of the SES scheme . The Bank covered part of the travel and relocation costs, workshops and seminars, but funded the entire cost of HR firm, impact evaluation, and audit . There were no project or physical contingencies . Financing The project was funded through a single -purpose World Bank trust fund -- the Trust Fund for Liberia (TFLIB) grant -- financed out of IBRD net income in the amount of US$ 2.3 million. To address the initial funding gap, the number of SES recruits was reduced to be consistent with the available resources . The project was restructured on December 29, 2010 and US$60,000 were reallocated from Component A to Component B of the project to cover the gap in funding for monitoring the performance of SES professionals, who were widely spread across the country . The closing date was also extended from December 31, 2010 to June 30, 2011 to compensate for the delays in implementation caused by weak project management capacity at the beginning of the project . Other donors providing for the project were UNDP (US$1 million), USA (US$1 million), Humanity United (US$1 million) and Greece (US$0.1 million). There was a funding gap of US$2.25. The ICR does not mention how much funding was actually provided by each donor by project closure and whether the funding gap was filled or not. Donor Contribution In addition to the World Bank's contribution of $ 2.3m, the SES project was also financed by the UNDP ($1 million); USAID ($1 million); Humanity United ($1 million); and Greece ($0.1 million). Borrower Contribution The Task Team leader informed IEG that the Borrower's contribution was in kind and included office space, workstations, fuel, vehicles, and telephone cards . Each of the 15 County Development Officers hired through the SES was issued new 4x4 wheel drive vehicles for use in the counties . Web-based articles, however, suggest that the Borrower faced some difficulties in paying for vehicles and fuel for these Country Development Officers. These reports could not be verified by information provided in the ICR or project files . Dates The project was approved on October 12, 2007, and became effective on th e same date. The project’s closing date was extended from December 31, 2010 to June 30, 2011 on December 29, 2010. The project closed as per revised schedule on June 30, 2011. There were no major deviations from planned dates during the course of the project. A mid-term review was carried out in March 2010 by an independent firm . 3. Relevance of Objectives & Design: a. Relevance of Objectives: The project was consistent with Liberia ’s development priorities at the time of appraisal, and specifically supported the third pillar of the 2006-8 interim Poverty Reduction Strategy Paper (iPRSP) -- strengthening governance and the rule of law. Under this pillar, the iPRSP (p. 63) stated that the Government would focus “on creating operational and institutional capacities for public sector management .� It continues to be relevant to Liberia ’s latest development priorities as spelled out in the 2008-11 PRSP, which maintained “Strengthening Governance and the Rule of Law� as a key pillar (ICR, p. 11), It is also consistent with the Bank ’s FY09-FY11 Country Assistance Strategy, recently extended till FY 12. The first strategic theme of the CAS is rebuilding core state functions and institutions (ICR, p. 11). While the objectives around building state capacity were appropriate, short and long term goals should have been better delineated and prioritized . The former would have been consistent pillar 1 of the overall country's strategy, and yet not too ambitious given the prevailing country conditions . Given that the project's longer term objectives were retained, it would have been important to balance state capacity building with an emphasis on "legitimacy...in society" (OECD DAC, 2011 Fragile States Principles Monitoring Survey of Liberia, p . 28). Based on the above assessment, the relevance of the project objectives is rated as substantial . b. Relevance of Design: The project components were consistent with the shorter term objective of rapidly building a cadre of skilled personnel to improve policymaking . The merit-based recruitment of Liberian expatriates rather than foreign experts was a pragmatic and innovative response to prevailing country conditions -- one that carries important lessons for other post-conflict, fragile states. Notwithstanding these strengths, this review did find some design weaknesses, primarily related to sustainability and the longer term objective of restructuring and reforming ministries. These are as follows: As discussed more in detail in Section 10 of this ICRR, monitoring and evaluation arrangements -- an essential element if the SES was to be integrated into the Liberian civil service -- were weak. The project made overly optimistic assumptions about the sustainability of a donor -financed SES wage bill and allowances; the impact of rapid wage decompression vis -a-vis other cadres (in some cases, 20:1 decompression ratios between SES and lower cadres ); and the relative size and ability of the SES to retain skilled professionals, given the competition for skill staff from the donor sector . Experience in other countries suggests that the recruitment of highly paid elite cadres does not alone result in agency level improvements. To align individual and institutional incentives, other cou ntries have used challenge grants that provide incentives -- in the form of capacity building funds and salary enhancements -- to Ministries, Departments, and Agencies to undertake performance improvement and re -engineering efforts (for example, in Tanzania) in a coordinated fashion. The design of the SES project does not include such a mechanism that would link recruitment of elite cadres to agency -level efforts (including technical assistance, capacity building, and policy initiatives funded under other operations ). Neither at design nor during implementation did the project include an plan for structurally embedding the SES within the civil service. As noted in Section 4 and 7 of this review, while various plans and concepts have been developed for public sector modernization, the GOL is "still unable to take the initiative to absorb the SES participants into the Civil Service " (p. 19 of the ICR). Based on the above assessment, the relevance of the project objectives is rated as modest . 4. Achievement of Objectives (Efficacy): As noted in Section 2a, this review assessed the efficacy of the project ’s short and longer term objectives . Short-term: Improving Policymaking and Service Delivery through SES Recruitment and Participation Overall, 80 percent of SES staff received positive assessments of their performance through annual contract reviews, well above the target of 75 per cent. This result is indicative of the benefits of merit -based SES recruitment. About half (48) of the SES Executives were placed in MDAs strategically linked to Liberia ’s Poverty Reduction Strategy. Substantial policy making activity was carried out by these MDAs, as shown in the ICR (see Table 3, pages 12 and 13 for a full list). Inter alia, SES executives contributed to the Implementation of the debt recovery plan leading to HIPC completion and debt waiver by Paris Club, the finalization of the National Employment Policy; and of the National Capacity Development Strategy; the development of the M&E framework for the Liberia Agriculture sector Investment Plan; the design of the national trade policy; the development of the national identification registry and of the Liberia Pay and Grade Reform Strategy; and the drafting of the Whistle Blower Protection Act. The Bertelsmann Transformation Index (BTI) offers a broader measure of improved effectiveness of policy making . The 2010 BTI ranks Liberia 31st out of 128 countries in its Management Index, ahead of all countries in West and Central Africa. The recruitment of SES cadres in key ministries did help make these immediate policy improvements (admittedly from a low base). However, the attribution of front-line service delivery improvements to the SES project is more difficult, and depends on a number of factors (for example, quality of sectoral institutions, performance of providers, demand for services, etc .) The ICR does not provide evidence to confirm this link . Overall, the achievement of the first objective is rated as substantial . Longer Term: Reform of Ministries and Departments to which Senior Executives were Assigned While the project’s longer term objective has yet to be achieved, some MDAs have developed restructuring plans . As noted in the ICR (p. 14), it was "anticipated that by the end of the project at least eight ministries and agencies would have developed and begun implementing a restructuring plan based on a redefined mandate, new organizational structure and matching staff plan .� By the end of the project, six MDAs had done so . In addition, several other ministries, not included in the original target list, have now initiated restructuring plan of their own and are at various stages of the process . These include: Education, Justice, Transport and Communication, Lands and Mines, National Investment Commission, Ministry of Gender, Vice President ‟s Office and Ministry of Foreign Affairs (ICR, p. 27). More evidence of consistency of methods used to define structures, conduct functional reviews, and align structures with budget and operational realities would have strengthened the ICR . Attribution to the deployment of SES cadres cannot be clearly established, especially given the number of other complementary public sector reform initiatives underway . More generally, the SES staff had not been integrated into the civil service, as originally envisaged, either at the end of 2010 or in 2011. The target date for SES integration has now been postponed till 2013. As noted in Section 3, the ICR (p. 19) concluded that “government seems unable to take the initiative to quickly absorb the SES participants into the Civil Service ....� The lack of progress on this front is explained in part by the absence of a clear strategy for institutionalizing the SES as an elite cadre in the civil service system (see Section 3b). Based on this assessment, the achievement of the second objective is rated as modest . Despite the above-mentioned limitations in terms of sustainability and unclear impact on service delivery, it is important to give appropriate weight to the short -term objective of staffing and recruiting the SES to improve policymaking as well as early efforts on MDA restructuring . According to the IEG's ICR guidelines for split sub-ratings, the overall assessment of efficacy is substantial . 5. Efficiency: While neither the PAD or ICR attempts a cost -benefit analysis (which would be admittedly difficult), the ICR does analyze the cost of the SES against several comparators : SES staff cost was about one third of the cost of consultants for similar positions in other Bank projects . The salaries paid for SES were lower than those paid by donors and banks and higher than those paid by International NGOs, Public Corporations and the GOL, for all job bands . The majority of SES professionals hold a master ’s degree. The ICR also points to the fact that non -salary expenses were kept to a minimum (only 10 percent). Although available evidence is limited, the program appeared to have provided a more viable and cost effective alternative to perpetual reliance on short -term, international consultants, advisers, and technical experts . Efficiency is therefore ranked as substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project offered a pragmatic, innovative, and efficient way of using Liberian expatriates rather than international consultants and therefore achieved tangible, short -term results of recruiting qualified SES cadres and improving policy making. While the relevance of objectives was substantial, the ambition of the project's longer term objective of reforming ministries was not matched by the design of appropriate incentive mechanisms to link SES recruitment to specific agency level actions and results . Assessed as modest, the design also needed to address financial dependency risks . These design issues were in turn reflected in the modest achievement of the longer term objective of reforming ministries and agencies . Nevertheless, efficiency is assessed as substantial since the project served as a viable and cost effective alternative to short -term reliance on expatriates. Taken together, and in keeping with the IEG/OPCS Harmonized Evaluation Criteria, the project ’s overall outcome is assessed as moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: IEG identified four sets of risk factors . First, there are financial viability risks associated with dependency on external funding of the recurrent costs of an elite GOL cadre . Absent a viable medium-term financing strategy, the SES recruitment and retention can be threate ned. The second set of risks relates to institutional sustainability . Without a viable plan, with early time -bound commitments to structurally embed the SES within the civil service, the GOL will likely face bureaucratic resistance due vertical and horizontal inequities (for example, due to rapid wage decompression and perceptions at the senior levels of horizontal inequities ). Third are incentive risks that result from weak linkages between individual SES deployments and agency level plans for restructuring and performance improvement. Finally, there are political risks to a program is closely associated with the current administration and may be in jeopardy in the event of a change of government . A number of technocratic initiatives are underway to mitigate these risks (for instance, an exit strategy, slightly postponed, to integrate the SES into the civil service; various pay and employment plans and strategies, and ongoing plans for public sector modernization ). These initiatives -- and design elements discussed in Section 3b -- needed to be implemented in a timely manner to ensure that SES objectives were "locked in" and the risks of reversal mitigated . As noted in Section 4 (and in the ICR, pl 19), the GOL continues to face challenges on this front . Risk to development outcome is therefore rated as significant . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project represents a pragmatic and novel Bank response to challenging country conditions . The recruitment of expatriates may offer useful lessons for other fragile and conflict affected states . The project was also relatively well coordinated with other donors (albeit with somewhat implementation arrangements with the UNDP) and closely aligned to the priorities of the iPRSP . Yet, given the design concerns presented in Section 3(b), as well as inadequate attention to sustainability risks, and the weaknesses in the M&E of an important pilot program such as this, the ICR review found Bank performance at entry to be moderately satisfactory . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The quality of supervision rested in the Bank's ability to proactively identify and resolve potential threats, for instance, by recruiting a management consultancy to remedy the poor performance on the part of the project coordinator hired by UNDP. However, these favorable aspects were tempered by one shortcoming -- real and perceived delays in approving disbursements that affected the timeliness of SES salary payments . Complaints of delayed payment of SES salaries have posed a reputational risk for the Bank . Some of these instances were made public in the Liberian press, which reported delays of up to 4 months. According to the comments by the Borrower in the ICR (p. 41), these delays were “due to delays in outing payment requests in Client Connections and securing “no objection� on time for payment.� On balance, this review concurs that Bank performance on supervision was satisfactory . Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government showed strong commitment to the project, but was slow in providing adequate staffing, did not set up an adequate M&E systems that could guide decision making, and did not pay adequate attention to the sustainability of the SES scheme once donor support would cease . All these weaknesses have been identified in the ICR (p. 23). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The initial ISR mentioned that the management of the SES program at the Civil Service Agency had been “very weak�, an issue that was progressively addressed through technical assistance . As a result, the Agency met most performance criteria. That said, it was delayed in submitting reports, missed some key milestones, and did not focus early enough on sustainability . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: This review concurs with the ICR (page 8) that “the monitoring and evaluation component was not properly designed to be able to capture the contribution of the project to the PDO ... Measurement indicators were mostly output based and not outcome based .� In addition, M&E was made more difficult by the multiple -- at times, different -- versions of the PDO presented in the original project document, the Grant Agreement, and the project's results framework. Furthermore, some outcomes did not identify corresponding measures . b. M&E Implementation: As noted in Section 8, implementation of M&E was mixed. As noted in the ICR (p.9), the M&E function “was not performed satisfactorily and when performed, record keeping was not adequate . Supervisors were not diligent in collecting data on employee performance .� As a result, reports were inconsistently filed and tended to be incomplete. c. M&E Utilization: The Borrower noted in its comments to the ICR (p. 41) that “supervision of SES professionals and staff has not been as effective as expected and required . “ Most supervisors in the Ministries, Departments, and Agencies did not provide the requested guidance to develop monthly, quarterly and annual work plans . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: The project, which consisted of capacity building activities, was classified as Category C, and the EPP (p. 15) confirms that no safeguards policies were triggered . b. Fiduciary Compliance: The ICR substantiates that financial management was highly satisfactory (p. 10). No issue emerged in financial management, or procurement, and all audits indicated proper compliance with Bank fiduciary requirements . c. Unintended Impacts (positive or negative): Unintended impacts are not evident . d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Significant Financial dependency and institutional Outcome : sustainability risks, coordination risks, and perceived politicization risks . Bank Performance : Satisfactory Moderately Shortcomings in design and the Satisfactory timeliness of SES payments due to disbursement delays. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The main lessons that can be drawn from the experience of this project and the ICR review are : Simple projects with focused short -term objectives are lower risk and can deliver higher impact results in low capacity, fragile settings . The SES operation was simple with two straightforward components and a small budget. In principle, it should serve as a building block for more sophisticated institution building measures. A project aimed at providing quick -surge capacity from expatriates or the private sector should pay early attention to issues of sustainability, knowledge transfer, and the factors needed to deliver longer term improvements in institutional performance . The still unproven sustainability of the SES is part of a more general problem with diaspora projects as emphasized by a 2011 study by the African Development Bank (AfDB, The Use of Diasporas in Government Functions in Post -Conflict and Fragile Situations , available here). This risk entails “little knowledge transfer....as a result of deploying only a few experts from the Diaspora to their countries of origin .� Strong M&E systems are necessary, particularly in pilot initiatives relating to capacity development and institution building. These should be designed properly and the results of the final assessment should always be included in the ICR or their exclusion explained . The reputation of programs should be monitored and claims made should receive adequate follow -up, even if they are made on-line rather than through more traditional channels . A number of other lessons are mentioned in the ICR . These could be pared down and focused on a key point that emerges from the SES program experience -- institution building at the center of government, especially in post-conflict settings, offer a rare opportunity for the Bank to advice on and help implement a vision for the role and structure of the state that is fiscally sustainable and politically durable . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: This well-written ICR provides a candid assessment along with relevant data . The report is internally consistent and tries to focus on results . Lessons learned are thorough but could be better prioritized . a.Quality of ICR Rating : Satisfactory