46481 QUALITY OF SUPERVISION ASSESSMENT (QSA7) Summary Assessment Sheet COUNTRY Sierra Leone PROJECT SL-Inst Reform & Cap Bldg TAL (FY04) REGION AFR SECTOR Public Sector Governance TASK TEAM LEADER ZHOU MANAGER Helga W. Muller COUNTRY DIRECTOR Mats Karlsson LN/CR AMOUNT ($MIL) 25.12 LN/CR NUMBER H0860 PROJECT ID P078613 TTL HQ-BASED Yes DATE APPROVED 01/15/2004 DATE EFFECTIVE 07/06/2004 PERSON(s) INTERVIEWED Yongmei Zhou; Parminder P. S. Brar SECTORAL COVERAGE: Sub-natl govt admin (65%) LICUS (Yes/No) Yes Central govt admin (35%) NON DEDICATED MULTI-SECTORS No PREVIOUS QAG None (Yes/No) ASSESSMENT(s) SPVN BUDGET $000 FY05 FY06 (incl. TF) 139 279 WPA 150 440 ACTUAL AT RISK STATUS EEND FY04 END FY04 ND FY04 EEND FY05 END FY05 ND FY05 EEND FY06 END FY06 ND FY06 ISR: 2-Flag 2-Flag 2-Flag PANEL: 2-Flag 2-Flag 2-Flag EA CATEGORY B FIRST STAGE REVIEW: PANEL MEMBER(S) William R. Dillinger; Michael L. O. OBSERVER Alexander Kremer Stevens MODERATOR Padmanabha Hari Prasad SPECIALIZED REVIEWERS: 1. Poverty and Social Maria Concepcion J. Cruz 2. Environment Colin P. Rees 3. FM Douglas I. Graham 4. Procurement Elzbieta Sieminska OVERALL RATING OVERALL RATING 1 ASSESSMENT DATE 09/05/2006 QUALITY OF SUPERVISION ASSESSMENT (QSA7) COUNTRY: Sierra Leone PROJECT TITLE: SL-Inst Reform & Cap Bldg TAL (FY04) A. Overall assessment 1. The Panel assesses the supervision of this project as Highly Satisfactory (1) because of the Team's consistent display of flexibility, energy, creativity and responsiveness to implementation challenges in a complex post-conflict environment where a blue-print approach supervision is doomed to fail, and where many implementation challenges could not be anticipated in design. The Panel took into account the comments from the Team on the draft Assessment, and made modifications in the comments in A. Quality at Entry (b), 3.2(c), 3.3, and 3.4(a). B. Main problems encountered (when did they occur, and what were the main factors giving rise to them) 2. Prior to project effectiveness, it was known that the lack of capacity at all levels of government, the conflicting agendas of stakeholders and deeply entrenched corruption would make implementation unpredictable. C. Appropriateness and adequacy of actions taken by the Bank to resolve existing or potential problems 3. When difficulties arose and delays occurred, the Team moved quickly to assist local counterparts to overcome obstacles by shifting resources, building capacity in new ways, overcoming stakeholder opposition, and bringing to bear the decentralization experience of other countries. The Team saw its role as facilitating the local project team find solutions to implementation problems, and did this effectively on many occasions. D. Any Systemic Lessons 4. Projects with major institution-building objectives in post-conflict settings require a high level of Bank resources, which is seldom recognized sufficiently when country team priorities are set, and Bank budget funds allocated. The Task Manager, recognized the need for close supervision and implementation support, was successful in mobilizing a high level of resources partly from the country budget, through a supportive Country Director, but also from donors and elsewhere in the Bank. Thus, she was able to provide the intensity of supervision the project required. This required considerable effort on her part. Nevertheless, sufficient funding should have been planned at the outset. 5. The experience of the IRCB makes it clear that reliance on quantitative indicators to monitor progress is insufficient. These should be supplemented by qualitative discussion of progress under each major component to determine what is really happening. The Task Manager did this. E. Suggestions to the Task Team (List two or three factors that will require particular attention in the supervision strategy over the short/medium term to improve the prospects for achieving the project's development objectives and long-term sustainability). 6. The Panel offers four suggestions to the Team: (i) Keep on top of pay reform at both central and local levels. Accountable public financial management and well performing local government depend in the long run on establishing a rewards structure which allows all tiers of Government to attract, retain and motivate competent staff, and removes the necessity for civil servants to augment inadequate salaries in informal ways, such as absenteeism and petty corruption. Currently the excellent performance of project counterparts depends critically on the contract terms they enjoy, courtesy of the project, which provide remuneration at multiples of current civil service pay levels. Local council staff, including those seconded from the central government civil service, receive grossly deficient official salaries. Unless pay is improved (and the Panel recognizes this is a necessary but not sufficient condition), sustainable service delivery cannot be built, and the project will have failed in its institution building goal. (ii) Ensure there is substance to PFM regulatory reforms. With respect to the public financial management reforms, it will be important to ensure that the legal and regulatory frameworks are not only put in place but are implemented effectively. This will entail fundamental changes in the way Government is managed (such as making vote holders accountable for the public monies they are responsible for) and will need to be closely monitored by the Team. (iii) Monitor how civil society uses greater space to demand transparency and accountability. Both the new PFM legislation and the local government act mandate major improvements in transparency and provide opportunities for non-state actors to hold Government accountable. It will be important for the Team, as the project unfolds, to ensure that these opportunities for holding the public sector accountable are taken advantage of. (iv) Supporting training institutions. If the current dispute over the location and control of the DLC are not resolved, the Team should explore whether resources could be shifted to rehabilitating the Institute of Public Administration, which is now under new leadership, and locating the distance learning facilities there. Rejoinder from the Region QUALITY OF SUPERVISION ASSESSMENT (QSA7) COUNTRY: Sierra Leone PROJECT TITLE: SL-Inst Reform & Cap Bldg TAL (FY04) Major Issues Identified Recommendation for Panel Skills QUALITY OF SUPERVISION ASSESSMENT (QSA7) Development Objectives (Part A to be completed by the Task Team and Part B by the Review Panel) A. List the primary Development Objectives outlined in the project's document. The Task Team is requested to indicate the importance of the DOs as: H = High or M = Moderate or N/A = Not Applicable. Development Objectives i. Poverty Reduction NA Outcome: Interim Benchmark: ii. Structural and Sector Policy Reform NA Outcome: Interim Benchmark: iii. Private Sector Development NA Outcome: Interim Benchmark: iv. Institutional Development/Capacity Building H Outcome: PDO: Establishing a functioning local government system and improving the inclusiveness, transparency, and accountability of public financial management at all levels of Government. Performance indicators include: (a) 19 elected local councils are able to make development plans that respond to local priorities through a participatory process and are able to prepare a budget consistent with Section 67 of Local Government Act 2004. (b) 14 of the 19 elected local councils meet the transparency and the financial management accountability requirement as per Local Government Act 2004 (Section 107, 81, 105). (c) 14 of the 19 elected councils are able to deliver all services devolved to them at the levels of the year before devolution. (d) 90% donor funding is captured in the fiscal reports using Government budget classifications by program and object of expenditures. (e) 95% of Vote Controllers submit regular in-year and annual budget execution reports to MoF and the relevant minister on time. (f) In-year reports are published regularly for tracking poverty expenditure by function. (g) Public Procurement Law enacted, embracing the principles of transparency, accountability and efficiency. (h) 90% of funds provided for procurement are effectively utilized. Interim Benchmark: Please see the annex at the end of the questionnaire v. Human Development NA Outcome: Interim Benchmark: vi. Environmental Sustainability NA Outcome: Interim Benchmark: vii. Infrastructure Development NA Outcome: Interim Benchmark: viii. Macroeconomic Management NA Outcome: Interim Benchmark: ix. Other (specify) NA Outcome: Interim Benchmark: Comments (Please explain any material changes in DOs since project start): Target Values Outcome Indicators June-05 June-06 June-07 June-08 Number of elected local councils which are able to make 5 10 14 19 development plans that responds to local priorities through a participatory process. Number of elected local councils which are able to make 5 8 12 14 budget consistent with Section 67 of Local Government Act 2004. Number of elected local councils which were able to meet the 5 10 12 14 transparency and the financial management accountability requirement as per Local Government Act 2004 (Section 107, 81, 105) Number of elected local councils which were able to NA 8 12 14 complete the projects submitted in previous year's work plan Number of elected councils which were able to deliver all Too early to assess 14 14 14 services devolved to them at the levels of the year before devolution. Number of elected local councils receive a tied grant for each 19 19 19 19 devolved service, at least that amount necessary to continue the operation and maintenance of that service at the standard to which it was provided in the year prior to its devolution Percentage of donor funding captured in the fiscal reports 10% 25% 75% 95% using government budget classifications by program and object of expenditures Percentage of Vote Controllers submit regular in-year and 40% 60% 80% 95% annual budget execution reports to MoF and the relevant minister on time. Routine in-year reports are published for tracking poverty yes yes yes yes expenditure by function. Percentage of recorded transactions reconciled to bank 95% 95% 95% 95% accounts within one month. Public Procurement Law embracing the principles of Public Procurement Bill transparency, accountability and efficiency. embracing the principles of transparency, accountability and efficiency submitted to parliament and enacted. Percentage of reporting entities preparing procurement 20% 80% 90% 100% plans as part of the budget preparation process. Percentage of the funds provided for procurement effectively utilized. 50% 60% 75% 90% The DLC is fully operational and has become 25% cost 50% cost 100% cost financially viable. recover- recover- recovered ed ed QUALITY OF SUPERVISION ASSESSMENT (QSA7) B. Likelihood of the operation achieving the DO1 (Use a scale of: 1 = Highly Likely, 2 = Likely, 3 = Moderately Likely, 4 = Moderately Unlikely, 5= Unlikely, 6 = Highly Unlikely, or NA = Not Applicable). i. Poverty Reduction NA ii. Structural and Sector Policy Reform NA iii. Private Sector Development NA iv. Institutional Development/Capacity Building 2 v. Human Development NA vi. Environmental Sustainability NA vii. Infrastructure Development NA viii. Macroeconomic Management NA ix. Other (specify) NA Comments: (Panelists are requested to identify the factors that affect the rating of likelihood of meeting the DO, comment on whether the key indicators identified by the task team provide direct measures of the expected outcomes, and indicate whether the stated DOs are consistent with the CAS and the legal agreement.) The likelihood of achieving the development objective of a functioning local government system and more inclusive, transparent, and accountable public financial management is judged to be good (2). The enactment of new laws (local Government, financial management, procurement),the creation of new supervisory bodies, and the holding of local Government elections and the launching of capacity building programs signify major shifts in the structure and process of Government. C. Sustainability: Panel's judgment of the likelihood that the operation's results will be sustainable in the longer- term (Panelists 3 should choose one of the following categories: 1 = Highly Likely, 2 = Likely, 3 = Moderately Likely, 4 = Moderately Unlikely, 5 = Unlikely, 6 = Highly Unlikely, or NA = Not Applicable). Comments: The Panel's assessment of sustainability is more cautious and is rated (3) moderately likely. This is based not on any deficiencies in project design (few) nor of supervision (commendable) but on the complex post conflict governance environment in Sierra Leone which entails major risks to the project's sustainability. These risks arise from the loss of public sector capacity, the endemic nature of rent-seeking, and security uncertainty. Particular risks to sustainable local Government capacity building include elite capture, patronage based employment, delays in fiscal transfers, MDA opposition to decentralization, and the inability of councils to generate to attract, retain, and motivate competent staff. 1Either as originally stated, or as revised if there has been Board-approved restructuring for exogenous reasons. QUALITY OF SUPERVISION ASSESSMENT (QSA7) Summary Assessment Sheet Assessment Rating 1 = Highly Satisfactory 2 = Satisfactory 3 = Moderately Satisfactory 4 = Moderately Unsatisfactory 5 = Unsatisfactory 6 = Highly Unsatisfactory NA = Not Applicable 1. Focus on Development Effectiveness 1 2. Supervision of Fiduciary/Safeguard Aspects 2 3. Adequacy of Supervision Inputs and Processes 1 4. Candor and Quality of ISR 1 OVERALL ASSESSMENT 1 The overall assessment is not an average of the assessments of the constituent elements of supervision. Instead, the panel should use its judgment in weighing the relative importance of each given the country and the project context. In making the assessment, the panel should consider the importance of each category, and within each category, the various questions, to supervision quality. QUALITY OF SUPERVISION ASSESSMENT (QSA7) Context: A. Quality at Entry (a) Assessment of project design at entry 2 i) For fragile states (LICUS), what was the scenario at entry (please check appropriate box) Post conflict recovery or political transition ii) For fragile states (LICUS), how much did the above scenario situation change during the project supervision period? (please check appropriate box) improve Comments: Since the civil war ended in 2002, Sierra Leone has remained politically stable. National elections were held in 2002 and will be held again in 2007. The first local elections in thirty years were successfully conducted in 2004, and for the first time since 1972, district councils were formed. Macroeconomic policy has been generally sound, and, together with significant aid inflows supporting projects and general imposts, the economy has responded well. Recent data from the Bank of Sierra Leone show continued high growth in agriculture, manufacturing, construction and service, Sierra Leone, however, remains one of the poorest countries in the world, with very low human development indicator scores. Large numbers of professionals and technical staff left the country during the turmoil, and the civil service and its institutions were largely destroyed. (b) Quality of PAD's results framework (including intermediate performance indicators) and readiness of implementation 2 arrangements for M&E i) Extent to which the planned outcomes and key indicators in the results framework are precise and relevant to the activities 1 undertaken by the project. ii) Extent to which the planned outcomes in the results framework are linked with the outcomes/objectives in the CAS. 2 iii) Extent to which there are adequate arrangements linking project results framework to the borrower's monitoring and evaluation 3 system. iv) Quality of baseline data 2 Comments: (Please indicate whether actions were taken or should have been taken to improve the results framework. Please assess the quality of the baseline data to support performance or results monitoring, especially Tier II indicators for IDA countries.) i) Planned outcomes and key indicators in the results framework are extremely precise and relevant to the activities undertaken by the project. If anything, the Team may have erred on the side of excessive precision. The PAD's results framework lists 16 separate outcome indicators and numerous and somewhat complex intermediate indicators. The outcome indicators were systematically tracked during supervision. The Borrower's system uses the same indicators which are reported on a regular basis. ii) The planned outcomes in the results framework are fully consistent with the objectives of the Transitional Support Strategy (which preceded the current CAS). The TSS focused on institutional reform and capacity building down to the district level. The planned outcomes are also consistent with the current CAS, which, drawing on the TSS, defines governance, decentralization and public financial management a strategic priority. iii) The borrower's monitoring and evaluation system, is extremely weak. The project does include funding to strengthen this function. Effectively, the Government's monitoring of PFM modernization and decentralization is through the M&E conducted under this project. iv) Baseline values were based on a HIPC assessment in 2004, at which time most of the values were nil--systems were not in place; reforms had not been adopted. Attempts to track progress in quantitative terms were largely dropped during supervision, in favor of more meaningful qualitative descriptions. The Panel supports this pragmatic approach to monitoring. (c) Borrower commitment to the Project 3 Comments: Borrower commitment to the project has proved mixed. While decentralization fits into the Government's overall strategy for maintaining political stability and building legitimacy, the devolution of power was opposed by certain sectoral ministers (particularly the Minister of Education) and by traditional chiefs, who saw the re-establishment of local Government as a threat to their own authority. Borrower commitment to the project's PFM modernization goals appears, at face value, strong, but the test will be when the time comes to implement the new financial management and procurement legal frameworks. This will require rewarding staff who exercise good stewardship and sanctioning those who continue to waste resources. Procurement reforms will require ministers to step back from direct contract award, and ensure that their officials follow open, competitive tendering. This will be difficult under present civil service incentive structures. (d) Readiness of implementation at approval 2 Comments: While the project required extensive supervision, the starting conditions for implementation were in place by Board approval or by effectiveness (which followed two months later). These included the preparation of a project implementation manual, a project financial management manual, and an operational manual for the proposed local Government development grants. (e) Overall implementation performance prior to FY05 NA Comments: Not applicable--the project became effective in FY05. (f) Extent to which the problems encountered during implementation have been identified at entry 2 Comments: (Please evaluate the quality of the risk assessments and mitigation measures.) Several of the overarching problems encountered during implementation--lack of sustained political commitment, weak implementation capacity, and corrupt and vested interests impeding structural reforms--were identified at entry. In addressing these problems, the Team relied more heavily on intensive supervision than originally proposed in the risk matrix. Certain problems affecting the project's sustainability were not identified at entry--particularly the difficulty of recruiting qualified staff (owing to the low salaries offered in most of the public sector) and the Government's reluctance to fully fund transfers to local government. These problems were, nevertheless, recognized early in supervision. To begin to address the former, the project financed a public sector wage study, which was well executed by external consultants.. To address the latter, the Bank will include full-funding of intergovernmental transfers as a disbursement condition in its next adjustment operation. QUALITY OF SUPERVISION ASSESSMENT (QSA7) Context: B. Compliance with safeguard policies Please mark below the applicability of and compliance with safeguard policies: (Yes, if applicable and fully complied with) Policy Applicability Compliance (Prior to FY05) i. Environmental Assessment (OP 4.01) Yes Yes ii. Natural Habitats (OP 4.04) No NA iii. Forestry (OP 4.36) No NA iv. Pest Management (OP 4.09) No NA v. Cultural Property (OPN 11.03) No NA vi. Indigenous Peoples (OP 4.10) No NA vii. Involuntary Resettlement (OP 4.12) No NA viii. Safety of Dams (OP 4.37) No NA ix. Projects on International Waterways (OP 7.50) No NA x. Projects in Disputed Areas (OP 7.60) No NA Comments: C. In light of answers to questions under A and B, what should have been the supervision strategy/focus during FY05-06? Comments: The Panel considers the supervision strategy actually followed in FY05-06 to be correct. This focused on building support for the reforms (through the implementation of Rapid Results Initiatives in the districts, for example ) and responding creatively to implementation challenges as they arose (by, for example, expanding the pool of RRI coaches and assigning one to each district and mounting workshops to explore and reach agreement on the role of traditional chiefs in local government .This was an appropriate response but time consuming and expensive. The Team also ensured that counterparts had access to good practice lessons from other decentralizing countries by, for example, arranging for the Director of Decentralization of Uganda to be appointed as a capacity building advisor to the project. QUALITY OF SUPERVISION ASSESSMENT (QSA7) Assessment Rating 1 = Highly Satisfactory 2 = Satisfactory 3 = Moderately Satisfactory 4 = Moderately Unsatisfactory 5 = Unsatisfactory 6 = Highly Unsatisfactory NA = Not Applicable 1. Focus on Development Effectiveness 1 1.1 Identification and Assessment of Problems 1 a) Timely identification and assessment of implementation problems? 1 b) Timely identification and assessment of possible threats to DO? 1 Comments: (please identify the implementation problems and threats to DO.) Very timely in being on top of problems. 1.2 Focus on Sustainability (extent to which the supervision mission paid attention to this aspect) 1 a) Borrower and Stakeholder Ownership 1 b) Technical Assistance, Training and Capacity Building 1 c) Evidence that project supervision showed: 1 i) appropriate approach to building state institutions (capacity and accountability) 1 ii) effectiveness of multi-donor approaches 1 iii) awareness of conflict or peace-building impact on political and security dynamics 1 Comments: a) There is strong overall ownership of the PFM modernization and local government capacity building goals of the project, and reversal in broad terms is unlikely. There are significant risks, however, that conflicts among stakeholders may threaten individual components of the reform (e.g., devolution of education). The pervasive disfunctionality of the public sector suggests that some changes may be more in form than in substance. And the GoSL will have to confront the need to improve the rewards structure, both at local and central government, if the former is to deliver mandated services efficiently and effectively, and the new PFM regulatory framework is to gain traction across the whole of government. Currently, other than those on special contract terms, mostly donor funded at multiples of base pay, civil servants are paid a fraction of what is needed to recruit, retain and motivate competent professionals, technicians and administrators, and break habits of informal salary augmentation and systemic petty corruption. This requires taking the pay study undertaken by the project, and developing it into a medium term PAR reform strategy, consistent with the MTEF now being developed. b) The technical assistance and training components are well specified in the project document but their impact and sustainability will depend upon public sector pay reform, so that staff trained can be retained and productively employed by central and local government. ci) Project design supports the establishment of a sound framework for local government and public financial management, and the putting in place of transparent reporting and accounting mechanisms. Supervision has appropriately supported these fundamentals of capacity building. In a post conflict setting, getting the basics of PFM in place is an immediate priority. The rapid transfer of mandates to local councils is a calculated risk given the huge capacity gaps, but justified in governance terms. Supervision has focused on moving forward the drafting and approval of new laws and regulations, and building capacity. cii) Supervision has been highly collaborative, with close involvement of the main donors supporting capacity building, DFID, EU, and UNDP. This has enabled the TM to leverage additional funds for supervision, and has resulted in the recent commitment of $25 million in trust funds by DFID and EU to further support project activities. ciii) Origins of the project lie in the GOSL's awareness of the need, after conflict, to create and empower representative local governments throughout Sierra Leone. This is not just to deliver primary services but also to re-establish the legitimacy of national government and achieve country wide security goals. The supervision team has been fully aware and supportive of this broader purpose. 1.3 Actions Taken and Follow-Up 1 a) Appropriateness of Advice and Proposed Solutions to the Borrower 1 b) Appropriateness and Speed of Bank follow-up action 2 c) Impact and effectiveness of Bank actions 2 d) Quality and timeliness of Mid-Term-Review (if it took place during FY05-06)? NA e) Quality and timeliness of Restructuring Plan (if it took place during FY05-06)? NA Comments: (Please discuss the effectiveness of project restructuring, if applicable, as a supervision tool.) The Team was prompt and effective in its follow up actions, responding to challenges and supporting the local implementation team. A case in point has been the role of traditional chiefs, overlooked in project design. When their potential opposition to elected councils arose during implementation, the Team facilitated workshops leading to an acceptable clarification of roles--the Makeni Agreement. 1.4 Performance Monitoring 2 a) Extent to which the task team made use of the PDO and intermediate outcome indicators to assess the project's 2 implementation? b) Extent to which the indicators (both quantitative and qualitative) have been used to identify and address potential obstacles to 2 the achievement of the DO (attention to outcomes)? Comments: a) IOs accurately reflect the principal activities under PFM and decentralization, which are supported by the project and have been used by the Team, as described in the supervision reports. b) Once the quantitative indicators (identified in the PAD) were converted into qualitative indicators (as monitored in the ISRs) they became more useful, although still too detailed. The Panel has the sense that the TTL and her Team were well aware of the project's overall development objectives throughout project supervision, and did not require PDOs and intermediate outcome indicators to identify potential problems. In fact, the highly detailed intermediate outcome indicators might have prevented the Team from responding flexibly to unanticipated problems if they had been the sole basis for project monitoring. 1.5 Effective use of the CPPR or other venues for portfolio review with the borrower to resolve problems affecting the project, 2 including cross-sectoral issues (e.g. governance, transparency)? Comments: There is always a risk in decentralization projects that central government MDAs resist the loss of powers, and hold back resources or otherwise frustrate the transfer of responsibilities and the means to discharge them. Inadvertently, Bank sector operations can also obstruct decentralization if the responsible Bank staff fail to appreciate how roles have changed, and reinforce the status quo. The Team effectively used the CPPR and other venues for portfolio review to explain the project to Bank country staff and impress on them the need to align sector operations with the decentralized framework for service delivery. This is now happening. QUALITY OF SUPERVISION ASSESSMENT (QSA7) 2. Supervision of Fiduciary/Safeguard Aspects 2 2.1 Procurement e.g., In the context of the project's procurement risk profile: appropriate level and adequacy of procurement 2 inputs; timeliness of advice and follow-up; adequacy of support to borrower. Comments: The Procurement Annex of PAD is written as per the required standard and is satisfactory with regard to the procurement arrangements and its clarity. However, transparency and corruption are identified in Project preparation as major issues in the country.There are at least two important papers dealing with these issues and providing the recommendations for improvement and building the anticorruption environment. These are "Issues Paper on Public Procurement" and "Report on Governance and Corruption Survey". The Procurement Annex should embrace these topics in more detail by either at least including the recommendations provided in these Reports which are relevant to the project activities and arrangements or by the reference to these recommendations. More enhanced chapter on procurement risks and a suitable action plan would also be expected since the procurement risk is rated high. Given the nature of the project which provides technical support on institutional reform and includes components on PFM and Public Procurement Reform, the collaboration of FM and procurement is important.The Project supports establishing the Integrated Financial Management Information System (IFMIS). The supervision of the Project is intensive. Senior procurement staff is assigned to the Project. Though the Procurement staff do not participate in all mission, the issues of procurement reform, implementation of which is identified as one of the outcome indicators, is monitored by the Team regularly. Based on the clarification provided by the Team, every supervision mission includes a procurement post review, and the recommendations are submitted to the Borrower and followed up. In addition, PAD requires that Procurement Audits and Contract Management/Expenditure Reports are prepared by the Borrower annually. The Team informed that during their last mission the Bank's Team reminded IRCBP of the need for an annual Procurement Audit. A TOR has been agreed upon and advertisement has been issued. The Audit is expected to complete before our Mid-Term Review in Dec 2006. 2.2 Financial Management e.g., In the context of the project's FM risk profile: appropriate level and adequacy of FM inputs; 2 timeliness of advice and follow-up; adequacy of support to borrower. Comments: The project became effective on July 6, 2004, just at the beginning of the review period. Prior to effectiveness, the FMS participated in a launch mission in May/June 2004. Conditions of effectiveness had included appointment of a qualified auditor and adoption of the financial management manual. During the period under review there was intensive FM supervision with the FMS participating in four missions: September/October 2004; in February 2005; in July 2005; and in June 2006. In the July 2005 mission a SOE review was undertaken and a report written on the results. The project FM risk was rated substantial at appraisal. The Team has rightly been concerned with the quality of financial management and audits at the local government level and, while this remains an issue, the Team has been consistent in following-up on this matter throughout the review period. The National Audit office has been auditing local government transactions and is currently finalizing its report. Unqualified audits by independent external auditors have been received on a timely basis and reviewed, as have FMRs. The FM rating was downgraded from satisfactory to marginally satisfactory in June 2006 due to persistent delays in replenishing the special account. Bank Management discussed the issue with the Minister of Finance, who committed to reducing delays. The FM supervision rating of satisfactory is based on FM supervision inputs which were commensurate with the assessed level of FM risk and appropriate follow-up on identified FM issues. 2.3 Legal Aspects e.g., Legal compliance; clarity/timeliness of advice 2 Comments: 2.4 Environmental Aspects e.g., Environmental impact, safeguard compliance 3 Comments: No Environment Specialist was involved during FY 05/06, the Team indicating that environmental issues were not considered a high priority by the GoSL; given the nature of the project interventions, the Team thought the risks small, reporting no problems with environmental management. However, in view of the nascent state of institutions in SL better use of specialist input would have been appropriate; indeed, the Team recognized that the appraisal process needs strengthening and indicated that in the near-term a field-based infrastructure advisor with DfID could be tasked with monitoring environmental safeguards. Training in environmental safeguards remains outstanding along with attention to a central database for safeguards application, etc. 2.5 Social Aspect e.g., Social impact, safeguard compliance 2 Comments: The quality of supervision of poverty, gender and social aspects is satisfactory. Even though the TT rated the poverty reduction DO as "NA," devolving responsibility for delivery of services at the local level is anticipated to have positive impacts on the poor. For example, one of the project's KPIs is "published reports on tracking poverty expenditures by function." The Operational Manual for the Local Government Development Grant component provides guidance on ensuring inclusiveness, in addition to the standard safeguards compliance provisions. QUALITY OF SUPERVISION ASSESSMENT (QSA7) 3. Adequacy of Supervision Inputs and Processes 1 3.1 Staffing 2 a) Staff Continuity 2 b) Supervision Skill Mix (with special attention to fragile states and multi-sectoral projects) 2 c) Impact of decentralization on supervision quality 2 i) TTL or sector staff 2 ii) Procurement 2 iii) Financial Management 2 Comments:(please comment on impact of decentralization of sector and fiduciary staff on supervision quality and discuss adequacy of staff attention to various components of multi-sectoral operations.) The current TTL was responsible for project preparation and has remained responsible for project supervision. This has clearly contributed to the project's success. She is based in HQ. Procurement and financial management were conducted by field staff based in Ghana and Uganda. This has been a positive development. 3.2 Supervision Activities 1 a) If TTL is HQ-based, quality of mission preparation and effectiveness of time spent in the field 1 b) If TTL is field-based, quality of coordination with and support from HQ NA c) If there was joint supervision with co-financiers, quality of coordination and other supervision activities. 1 d) Adequacy of overall supervision mix from HQ, in-country, and neighboring country. 2 e) Effectiveness of Budget Use (including use of TFs) 2 Comments: a) The effectiveness of the supervision missions has been extremely high. Each mission appears to have focused on the key issues affecting the project's overall development objectives and has managed to reach agreement with the Government on specific, time- bound actions to rectify problems. These are documented in the thorough and clearly written aides memoire. Supervision missions also responded flexibly to events, adjusting the components of the project (and adding new ones using TF resources) as required. c) The project itself if financed exclusively by IDA. The other major donors in Sierra Leone--DFID and EU--were initially invited to participate in missions in order to improve coordination. This has proven highly successful. EU was sufficiently impressed with the project that it decided to drop its own local governance program and support the Bank's through a trust fund. The EU decision also triggered an offer by DFID to participate in a multi-donor trust fund totaling $25 million. d) Due to the small size of the Bank office in Freetown, the project has been supervised by staff from HQ and from field offices in Ghana and Uganda. The costs of supervising this project have been high, by Bank standards. Supervision costs in 2005 totaled $150,000. In 2006, they totaled $440,000. Of the latter figure, $147,000 was financed from trust funds. These costs are justified by the institution-building character of the project, the weakness of Sierra Leone's institutions, and the need to work closely with a wide range of stakeholders (including 19 separate local governments) to address implementation problems as they emerge. The substantial contribution of trust funds to the costs of supervision indicates the Team's success in leveraging Bank resources. The practice of allocating part of the Bank's trust fund administration fee directly to the supervision team should be practiced more widely. 3.3 Quality of supervision documentation including aide-memoire & management letters and follow-up 2 Comments: Supervision missions have been well documented. Each mission left a thorough and well written aide memoire, clearly identifying key issues and agreements. The project ISRs appear to be less useful, largely due to the constraints imposed by the ISR format. As noted earlier, the ISRs track progress on 16 output indicators. These were defined during project preparation and are still relevant, with minor adjustment of one indicator (on public FM reform) and achievement of the target on another (procurement law). The space allotted to conveying the extent of progress against these indictors is insufficient. As the project has adapted to changing circumstances, other evidence of progress (as discussed in the aides memoire) would be more useful. 3.4 Effectiveness of Relationships 1 a) With the Borrower 2 b) With donors and other stakeholders 1 Comments: a) The Team has had an effective relationship with the Government, despite uneven Government support for the project objectives. The aides memoire indicate that a combination of extensive direct discussions with Government officials and liaison with pressure groups has been effective. Threats (such as the cancellation of the DLC) have been less so. In its comments on the draft Panel report, the Team noted that in its view it has had a strong and satisfying relationship with the borrower, based on partnership and mutual commitment to learning. b) The relationship with DFID and UNDP has been highly productive. As noted earlier, both organizations have agreed to dovetail their local governance operations in Sierra Leone with the Bank's. 3.5 Management Inputs 2 a) Adequacy and speed of management attention and actions 2 Comments: Management comments were prepared by the country director (Mr. Karlsson) and sector manager (Ms. Mueller) except on one occasion when Ms. Mueller was traveling. (In that case, the comments were prepared by the acting sector manager.) The Panel is not aware of any problems in this area. b) Adequacy of supervision budget i) Overall budget Too Little Comments: (Please assess whether budgetary resources are appropriately distributed for multi-sectoral projects). While the BB budget allocation for FY05 meet about 95% of actual costs, the BB allocation for FY06 was sufficient to cover only 63% of actual expenditure. The difference was financed by trust funds. As the Panel believes the FY06 resources were used effectively, the overall initial allocation of BB resources appears to have been inadequate. QUALITY OF SUPERVISION ASSESSMENT (QSA7) 4. Candor and Quality of ISR (FY05-06) 1 4.1 Extent to which ratings for DO and IP reflect actual conditions? 1 Comments: The Panel believes the Team's ratings accurately reflect actual conditions. 4.2 Adequate explanation of DO and IP and of any change in ratings? 1 Comments: The progressive downgrading of ratings for the DLC are in line with GOSL failure thus far to implement this component of the project and is adequately explained in the ISRs. 4.3 Accuracy of ratings of project components and discretionary flags? 2 Comments: Satisfactory. 4.4 Quality and timeliness of data (including the intermediate outcome indicators) to support the key performance indicators. 2 Comments: Satisfactory. 4.5 ISR focus on issues and actions. 1 a) TTL Inputs 1 b) Management Comments 1 Comments: a) TTL summary of findings and recommended actions are well set out and to the point. Particularly helpful supporting documentation and the comprehensive aides memoire sent to the Government at the conclusion of each mission and the less formal BTORs prepared on return to Washington. With the ISRs, they give a very full picture of what is happening. b) Management comments have been timely and supportive. QUALITY OF SUPERVISION ASSESSMENT (QSA7) Risk Flags and DO/IP Status COUNTRY Sierra Leone PROJECT ID P078613 PROJECT TITLE SL-Inst Reform & Cap Bldg TAL (FY04) DO/IP/Risk Factor FY04 FY05 FY06 PSR Panel ISR Panel ISR Panel DO/GO S S S S S S IP S S S S S S Project Components DECENTRALIZATION AND S S S S S S CAPACITY BUILDING PUBLIC FINANCIAL S S S S S S MANAGEMENT REFORM DEVELOPMENT LEARNING S S U U MU MU CENTER IRCBP COORDINATION S S S S S S UNALLOCATED S S S S S S Discretionary Flags Financial Performance1 No No NA NA NA NA Counterpart Funds2 NA NA No No No No Financial Management No No No No No No Legal Covenants No No No No Yes Yes Monitoring and Evaluation* No No No No No No Project Management* No No No No No No Procurement No No No No No No Safeguards No No No No No No Critical Risk1 No No NA NA NA NA Sub-Total 0 0 0 0 1 1 Exogenous Flags Disbursements Delays* No No No No No No Effectiveness Delays* No No No No No No Country Environment* Yes Yes Yes Yes Yes Yes Country Record* Yes Yes Yes Yes No No Long-term Risk2* NA NA No No No No Sub-Total 2 2 2 2 1 1 Total Flags 2 2 2 2 2 2 Golden Flag No No No No No No Overall Risk Rating 2-Flag 2-Flag 2-Flag 2-Flag 2-Flag 2-Flag Comments: 1Old flags that were dropped in Jan '05. 2New Flags introduced in Jan '05. "Counterpart Funds" was formerly "Financial Performance Flag" and "Long-term Risk" was formerly "Critical Risk Flag." * Represents risk-flags applicable to Development Policy Lending operations as well.