ICRR 14058 Report Number : ICRR14058 IEG ICR Review Independent Evaluation Group 1. Project Data : Date Posted : 05/22/2013 Country : Comoros Is this Review for a Programmatic Series? Yes No Series ID : First Project ID : P117229 Appraisal Actual Project Name : Comoros US$M ): Project Costs (US$M): 3.00 3.13 Development Policy Grant L/C Number : Loan /Credit (US$M): Loan/ US$M ): 3.00 3.13 Sector Board : Poverty Reduction US$M ): Cofinancing (US$M): Cofinanciers : Board Approval Date : 06/01/2010 Closing Date : 06/30/2011 06/30/2011 Sector (s): Central government administration (100%) Theme (s): Public expenditure; financial management and procurement (70%); Administrative and civil service reform (20%); Other economic management (10%) Evaluator : Panel Reviewer : ICR Review Group : Coordinator : Thomas Bossuroy Clay Wescott Ismail Arslan IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Program Document (PD) (Annex 2, Policy Matrix) there are two objectives (and a number of sub-objectives): strengthening public financial management (sub-objectives: the quality of the budgetary process is improved: the budget is a tool for setting policy priorities; Improved budget implementation and monitoring of budget execution, comprehensive upgrading of public financial management tools, capacity and processes), and strengthening public sector efficiency and accountability (sub-objectives: improved wage bill management, improved deployment of civil service human resources ). Other statements of objectives in the PD are consistent with this formulation (p. i, 10), but the statement in the Policy Matrix is clearer .. b. If this is a single DPL operation (not part of a series), were the project objectives/ key associated outcome targets revised during implementation? No c. Policy Areas: The program addressed two broad policy areas (PD, pp. 12-15): (i) improving public financial management (PFM): it was assessed as extremely weak in the Comoros, with services poorly organized and outdated procedures. Revising the PFM system would contribute to aligning the budget to policy priorities, as well as setting up a fiscal arrangement critical for consolidating Union-island cooperation and ensuring stability . (ii) strengthening public sector efficiency and accountability: disorderly decentralization and duplication of governing bodies has created a bloated and poorly managed civil service. Fiscal sustainability requires better controlling the wage bill by reducing the staffing but also better managing recruitment and wages through a centralized personnel management unit equipped with a computerized management system . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The DPL was approved on June 1, 2010 for SDR 2 million (equivalent to US$3 million), fully disbursed October 12, 2010 on effectiveness, and closed on schedule on June 30, 2011. The difference between the appraisal and the actual amount is due to changes in the exchange rate . There was no co-financing. 3. Relevance of Objectives & Design: a. Relevance of Objectives: The program directly supported two of the six axes of the country ’s Poverty Reduction and Growth Strategy Paper (PRGSP) : (i) axis one, on economic stabilization and equitable growth, emphasizing improvements in government and fiscal operations; and (ii) axis three, on strengthening governance and social cohesion, focusing on improvements in core personnel management systems . The project objectives are also relevant to the Bank’s broader strategy in the Union of Comoros . The Interim Strategy Note for the period fiscal year (FY)10-12 is structured around two objectives – reducing social vulnerability and strengthening state capacity and accountability. The project aimed at contributing to this second objective . The overall relevance of objectives is assessed as high. high b. Relevance of Design: The choice of prior actions and triggers was well aligned with the project objectives (see Box 1, PD page 12). The results framework is sound and well articulated with the overarching objectives and the more specific objectives for each policy areas . For example, the prior actions of preparing a consolidated budget, quarterly budget execution reports, and initial steps in implementing the public financial management (PFM) strategy and action plan are logical steps towards improved budget preparation and execution . The triggers of preparing a new fiscal regulatory framework, inclusion of social spending information and public disclosure of budget execution reports, and improved capacity in budget organization and internal and external controls, all represent the next logical stages of PFM reform. Likewise, the prior actions of implementing a payroll system and issuing an administrative order reaffirming the pre-eminent role of the High Authority of Civil Service (HACS) are logical steps towards the objective of public sector efficiency and accountability, since there had been a worrying trend of unsustainable growth in the wage bill relative to GDP. Likewise, the triggers of continued use of the personnel management system, and public disclosure of admitted candidates are plausible, logical next steps of civil service reform . The limited scope of reforms was appropriate given the size of this project and in order to pursue a gradual reform approach in a challenging context . The overall relevance of design is assessed as high. high 4. Achievement of Objectives (Efficacy): In a broad sense, significant progress has been made by Comoros towards a more stable macro -economic environment and stronger governance through PFM and public sector reforms . After slow progress between mid-2010 and mid-2011, a committed government took bold moves to implement the reform agenda supported by donors. The fact that Comoros officially reached the Highly Indebted Poor Country completion point on December 20, 2012 is a compelling testimony that steady progress took place on key areas supported by the project, and the project objectives. Assessing the efficacy of the operation itself is elusive, as it was one component in a wide range of actions spanning many sectors. The project sets down a series of performance indicators, most of which are covered in the Public Expenditure and Financial Accountability (PEFA) assessment; yet unfortunately recent PEFA ratings were not available at the time of writing the ICR review, so that it is not possible to assess the efficacy against those indicators. Although attribution remains an issue, it is possible to examine the progress made towards the completion of the development objectives. 1. Strengthening public financial management The first sub-objective was to improve the budgetary process and make the budget a tool for setting policy priorities. Significant progress was made. The Public Financial Management Law (Loi des Operations Financières de l’État - LOFE) was approved by the National Assembly on June 14, 2012. The trigger of preparing and discussing an organic law prior to submission for approval, was exceeded . The new law modernizes the PFM legal framework, including aspects concerning budget preparation and inter -governmental relations. The second sub-objective related to budget implementation and monitoring of budget execution . The publication of quarterly budget execution reports began at end of the last quarter of 2011. However, the quality of these reports is limited: social expenditures are not reported and comparison to the budget is not possible for main administrative headings. Moreover, the reports have not been made available to the public . The trigger for continued support was partially met . The third sub-objective focused on upgrading Public Financial Management tools, capacity and processes . After June 2011, significant progress was made . The following pieces of legislation or regulation were approved : legislation on ministry personnel frameworks (organigram and staffing plans), a law adopting a new public procurement code consistent with the Common Market for Eastern and Southern Africa (COMESA), a decree to reform the Treasury and Public Accounting areas, a draft decree to establish the General Directorate of Financial Control (Direction Générale du Contrôle Financier), and a decree creating the Autorit é de Régulation des Marchés Publics and defining other important public procurement reform implementation issues . Overall, the efficacy for this broad policy objective is assessed as substantial . 2. Strengthening public sector efficiency and accountability The first sub-objective related to wage bill management . Significant progress was made towards an automated system for managing civil service human resources and payroll . From July 2011 to March 2012, civil service salaries were progressively incorporated in the newly installed computerized system - GISE. It now covers most of the administration, with the exception of the military . The second sub-objective consisted in improving the deployment of civil service human resources . Little progress was made and the trigger for continued support was not met . Occasionally, recruitment of “A� level staff at Union level followed competitive procedures, but competitive procedures are not followed at Island level, where an important share of that recruitment takes place . Overall, the efficacy of this broad policy objective is assessed as modest . 5. Efficiency (not applicable to DPLs): 6. Outcome: IEG is in agreement with the ICR outcome rating of moderately satisfactory, given that (i) the development objectives were highly relevant to the country ’s challenging circumstances, (iii) the relevance of design is high and (iii) the efficacy is rated significant for policy area 1 and modest for policy area 2. Successes included progress in PFM and civil service management . Shortcomings were: (i) lack of public disclosure of budget execution reports, (ii) lack of widespread use of competitive procedures to recruit staff at “A� level and disclosure to the public of their ranking, and (iii) delay in cleaning up the GISE personnel database on the basis of the 2011 census. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: As pointed out in the ICR, the political environment has been a risk factor in the Comoros in the past decades . Although the political scene has recently stabilized and the electoral timeline gives some breathing space, fundamental fragilities persist, which may threaten social peace if not addressed . In particular, imbalances in the distribution of power and financial flows between the three islands pose particular threats to some of the key policy areas addressed in this DPL . The lack of integration of fiscal administrations may be an obstacle to successful PFM reforms. The differences between the three islands in terms of civil service recruitment practices and salaries are likely to hamper progress in civil service management . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Quality at entry is deemed satisfactory . This single-tranche project was originally designed as a programmatic series of two DPLs with the second DPL left as a possibility . This option was abundantly discussed during project preparation and with OPCS . The rationale for a programmatic approach in a context of low capacity and high volatility was that a first operation should pragmatically target a small number of feasible steps, create buy -in and pave the way for more ambitious follow -up operations. This choice takes into account the weak capacities and political fragility of the country . But in order to allow flexibility in an unstable environment and avoid creating a rigid operational framework that would likely need to be restructured, a single-tranche operation was retained . The PD outlines the “possible� triggers for a follow-up operation. Thus, the present operation supports initial reform steps, with fairly modest, prior actions, This design is well adapted to a volatile context, and turned out to be an effective way to start an ambitious reform program . The second DPL was approved on Nov 29, 2012, and builds on the momentum and achievements of the first operation - but it was delayed such that both operations are technically stand -alone. There is adequate alignment with both the Government priorities and the broader reform program carried out with the IMF and other donors and aiming at the Highly -indebted Poor Country (HIPC) completion point. The project design is underpinned by the Bank ’s previous analytical work on public financial management and public sector efficiency in the Comoros in 2001 and 2005. The choice of a single-tranche programmatic operation is well adapted to the challenges of the country . Its linkage with a technical assistance operation, the Bank's Economic Governance Technical Assistance (EGTA) Project, approved on January 31, 2011 is relevant, in a context of low capacity in the administration . The close coordination with a related IMF project was useful for leveraging on the momentum for reforms and reaching a greater efficacy than what the modest DPL would have achieved just by itself . Weaknesses include the choice of indicators, both because of the excessive reliance on PEFA assessments (which are still not available as of May 2013) and because of the little relevance of certain indicators to the policy area in question (as appropriately discussed in the ICR page 15). at -Entry Rating : Quality -at- Satisfactory b. Quality of supervision: Supervision appears adequate, mostly thanks to the coordination with the EGTA project team and with the IMF. The integration of the project in the broader reform process supported by other donors allowed for an efficient monitoring of the project in spite of the absence of a country office in Comoros . It would have been useful for the ICR to discuss how the Bank handled the delays and lack of Government ’s commitment that were experienced from mid-2010 to mid-2011. Quality of Supervision Rating : Satisfactory Overall Bank Performance Rating : Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The completion of prior actions is a sign of commitment, but is a modest achievement given their limited reach. There were then significant lapses in the completion of further reforms such as the publication of the budget, the competitive recruitment of civil servants and the preparation of a civil service census . The “loss of reform momentum from mid-2010 to mid-2011� (ICR, p. 13) created delays and led to postponing the preparation of a second DPL. The High Authority for Civil Service, which was created in 2008 and meant to lead the reforms of public sector, has little leverage and insufficient capacities . However, the Government’s performance dramatically improved after mid -2011 such that major reforms were completed (see timeline of reforms in ICR, Table 2, p. 12) and the HIPC completion point was reached in December 2012. Government Performance Rating : Moderately Satisfactory b. Implementing Agency Performance: The Ministry of Finance was responsible for coordinating actions and monitoring reforms, and the High Authority for Civil Service was also in charge of leading the reforms of the public sector . Given the nature of the reforms, it is however difficult to isolate their performance from the Government ’s performance. IEG agrees with the ICR to give them the same rating . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: A set of indicators was identified as monitoring tools for the progress of the project . Baseline figures were collected and reported. But relying on the PEFA indicators was not suitable in a volatile environment . Due to considerable delays those indicators are still not available for follow -up such that the evaluation is left without robust evidence on the critical outcomes . Further, choices of indicators are sometimes debatable . For example, it is unclear why arrears monitoring and accumulation is an indicator for the timely production and availability of budget execution data . b. M&E Implementation: An Annual Report on PRGSP was produced by the Ministry of Finance toward the end of the project . This report was not specific to the EGTA project but rather pertained to a much broader reform program . c. M&E Utilization: There is no evidence that M&E data was used along the project to inform further actions . M&E Quality Rating : Modest 11. Other Issues a. Safeguards: No Environmental or Social Safeguards were triggered by the project . b. Fiduciary Compliance: The internal and external control environment for public resource use is not yet operational, so fiduciary risks continued to be high. c. Unintended Impacts (positive or negative): None. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Satisfactory The shortcomings in the quality at entry Satisfactory relate to the choice of indicators . In the context of this operation, IEG considers these as minor and rates the quality at entry as satisfactory. Both dimensions of the Bank performance are therefore rated satisfactory so the overall assessment is satisfactory . Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: IEG broadly agrees with the lessons drawn in the ICR : It is critical for such projects to create and maintain a strong link with TA activities . It accelerates the pace of activities, strengthens the Government ’s capacity and ownership and contributes to the sustainability of the reform agenda. It also creates economies of scale for monitoring and supervision . A close coordination with the IMF and other donors is also critical for creating and maintaining momentum for reforms. The pursuit of such a major objective as debt relief helps creating a favorable policy window . Progress indicators in fragile countries should ideally be flexible and reasonably simple to collect, such that monitoring can take place even in an uncertain environment . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is well structured, concise, informative and rigorous in its analyses of successes and failures . The lack of hard evidence on the completion of outcomes is partly attributable to the modest size and short duration of this project. It would have been useful for the team to provide preliminary ratings for PEFA scores at the time of ICR based on existing evidence . The ICR should also have discussed the nature of the Bank ’s engagement with the Government over the first year when the momentum for reform was low . a.Quality of ICR Rating : Satisfactory