Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) Report Number : ICRR0021056 1. Project Data Operation ID Operation Name P153084 MG - Resilience DPO Country Practice Area(Lead) Madagascar Macroeconomics, Trade and Investment L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IDA-57520 30-Dec-2016 55,000,000.00 Bank Approval Date Closing Date (Actual) 10-Dec-2015 30-Dec-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 55,000,000.00 0.00 Revised Commitment 55,000,000.00 0.00 Actual 54,642,284.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Mauricio Carrizosa Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Policy Areas a. Objectives The Program Document (PD, p. v)) articulated the objectives as follows : "(i) strengthening reporting and increasing availability of information relevant to assessing the effectiveness of public finance, and (ii) improving payroll management and consolidating accounts in line with the treasury single account principle. b. Were the program objectives/key associated outcome targets revised during implementation of the series? Page 1 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) --- c. Pillars/Policy Areas There were four policy areas: 1. Civil Service reform: To improve payroll management, this area covered elimination of ineligible workers from the payroll, recording of civil servant information, and reconciliation of payroll and civil service data bases. 2. State-owned enterprise reform: To increase fiscally relevant information on SOEs, this area covered publication of diesel consumption and electricity generation data by JIRAMA (the electricity and water company); and publication of audited accounts of Air Madagascar (the public airline company), together with appointment of a new manager. 　　 3. Tax expenditures: To improve transparency of tax expenditures this area covered publication of free zone benefit grantees; 　　 4. Budget and financial management: To increase information on the budget, this area covered consolidation of externally financed project accounts, submission of accounts to Parliament, and monthly publication of budget results. d. Comments on Program Cost, Financing, and Dates The DPO was approved by the Board on December 10, 2015 in the amount of $55.0 million equivalent. The DPOs was declared effective on December 28, 2015. It disbursed on December 30, 2015 and closed, as scheduled, on December 30, 2016. 3. Relevance of Objectives & Design a. Relevance of Objectives The operation’s objectives addressed five specific issues that the Government of Madagascar faced: (i) the high share (42.5 percent) of wages in the budget; (ii) the poor financial and operational results of the state- owned power and airline companies; (iii) large tax expenditures through export processing zone benefits; (iv) fragmentation of funds from externally financed projects in separate commercial bank accounts; and (v) delays in budget accounts and audits. Objectives are aligned with i) the Government’s 2015-19 National Development Plan, which includes programs on public finance and public administration respectively in its macroeconomic stability and governance pillars; ii) the WBG’s 2015 Systematic Country Diagnostic, which highlights Madagascar’s high tax expenditures, the large share of wages in public expenditure, the deterioration of public expenditure and financial accountability (PEFA) performance, and poor governance of Page 2 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) state-owned enterprises (SOEs); and iii) the FY17-FY21 Country Partnership Framework (presented to the Board after the DPO; see PD, para 53), specifically with its objective to improve accountability in the management of public resources. Rating Substantial b. Relevance of Design The objectives were clearly stated. However, results chains were weak, as outcomes in the results matrix reflected primarily outputs (e.g., submitting audited accounts to Parliament). Causal chains from inputs to outputs were straightforward in most cases and largely reflected the implementation of Government decisions supported by the operation. In one case, the result (payments to departed officials), was virtually the same as the reform (elimination of departed officials from the payroll). Accordingly, the operation largely lacked a substantive theory of change. A more programmatic design would have strengthened the operation in this regard. In November 2015, before the DPO was approved, Madagascar entered into an IMF Staff Monitored Program. Although an economic recovery that began in 2014 was weakening due to falling commodity prices, weather- related shocks, and structural weaknesses, macroeconomic policies had succeeded in maintaining economic stability and sustainability in 2015. 　 Both the IMF program and the operation faced risks of faltering reform implementation, economic confidence, donor support, and tourism earnings; of the need for additional transfers to cover losses in JIRAMA and Air Madagascar; of failure to secure all of the projected external financing in 2015; and of natural disasters and weather related shocks. These could have resulted in lower priority spending on investment and social development and lead to additional domestic arrears, and in the diversion of resources away from the DPO- supported reform program. IDA expected that momentum from strong reforms and other IDA support (e.g., TA) would help sustain implementation and that dialogue on enhancing resilience to disasters (e.g., rehabilitation of early warning systems) and a revival of support from development partners would help mitigate risks from disaster and weather shocks. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Page 3 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) Objective 1: Strengthening reporting and increasing availability of information relevant to assessing the effectiveness of public finance. Rationale Outcome indicators relevant to this objective cover reporting or information on the budget, SOEs, and tax expenditures. On budget reporting, the operation supported the Government’s submission of the 2009-11 backlog of audited accounts to parliament in 2015 (a prior action) and expected delays in submission to decline to at most 24 months (from a 2014 baseline of four years). This target was achieved with the 2015 accounts, submitted to Parliament in June, 2017. The operation also supported the Government’s publication of monthly budget execution reports no later than 60 days after the end of the relevant month, starting in May 2015 (a prior action) and expected that the delay in publications would decline to 6 weeks (from a baseline of "not publicly available"). The delay declined to 8 weeks, two weeks above the target. Other available Indices of progress on overall budget performance vary. The CPIA rating for public financial management improved from 2.0/6.0 in 2014 to 3.0/6.0 in 2016. The World Economic Forum measure of diversion of public funds declined from 2.5 in 2015 to 2.2 in 2017. The Open Budget Index of budget openness, a more relevant indicator for Objective 1, has only been measured for 2017. Its ratings of 34/100 for transparency, 9/100 for open participation, and 28/100 for budget oversight indicate poor budget openness. For example, on transparency, the index value reflects "Minimal Information Available". Accordingly, the progress in achieving the intended appears limited. 　 With regard to the reform of state-owned enterprises, the operation supported actions on the Government’s power and water utility (JIRAMA) and its airline company (Air Madagascar). JIRAMA began publishing diesel delivery and generation statistics (a prior action) and monthly updating of such statistics (from a 2014 baseline of "no information") and a reduction in the amount of diesel used form 0.16 L/kwh (2014) to 0.14 L/kwh were targeted. Both targets were met, with statistics updated monthly and the use of diesel reduced to 0.13/kwh, replaced by less expensive fuel oil. However, there is no convincing link between the result on diesel use and the reform on publication supported by the operation. Air Madagascar appointed a new general manager through a competitive process, and published its audited accounts for 2010-2013 on its website (a prior action). Publication delay was targeted to decline to six months after the end of the FY (from at least 23 months at the time of Board approval). The publication of the 2016 accounts, in November, 2017, suffered a delay of 11 months, five months after the target date. 　 With the aim of greater transparency concerning tax expenditures, the operation supported actions to disclose beneficiaries of free zone benefits. As a prior action, the Government published the list of Free Zone Regime beneficiaries. Enterprises receiving benefits under the Large Mining Investments Law (LMIL), the Export Processing Zone (EPZ), and by Council of Ministers decisions, were also expected to be publicly listed and the list updated annually (from a 2014 benchmark of no information available). The target was partially achieved, as the Economic Development Board published two of the three lists (EPZ and LMIL), whilst the list of beneficiaries from Council of Ministers decisions (managed by the Prime Minister’s Office) has not been published. 　 In sum, the operation contributed to reduced delays in publication of audited accounts, budget execution reports, and SOE statistics and accounts; and provided information on EPZ beneficiaries. However, achievements were comprised mainly of outputs and were limited in their significance and Page 4 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) comprehensiveness. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Objective 2: Improve Payroll management and consolidate accounts in line with the treasury single account principle. Rationale The outcome indicators relevant for this objective cover payroll management and consolidation of externally funded project accounts. On payroll management, it was anticipated that the number of civil servants without date of birth information would be reduced from 30,000 to zero, that payroll (Ministry of Finance and Budget, MFB) and civil service (Ministry of Civil Service, MCS) data bases would be reconciled every six months, and that payments to officials who had departed "high officer" positions would be stopped. By 2017, the number of civil servants without date of birth information had declined to 4,297 (compared to the target of zero). Payments to officials who had already left "high office" service were suspended, (thereby achieving the target of zero payments), resulting in savings of 0.1 percent of GDP by February 2017. The target of reconciling MFB payroll and MCS civil service was not achieved because MCS did not update its civil service roster (the ICR reports that "MCS was not able to secure financing to update the civil servants roster"). Payroll controls are being pursued instead by requiring all agencies to declare staff details for input into a new resource management software. The ICR reports that a first round of declarations led to the discovery that 5,000 non-declared individuals were receiving pay. 　 The operation supported a government decree to consolidate externally funded project accounts. Government accounts for externally financed projects were transferred from commercial banks to the Single Treasury Account at the Central Bank (prior action). It was expected that the share of accounts transferred would increase from zero percent (2014) to 90 percent, with any remaining accounts to be closed within a deadline. By 2017, 87.4 percent of the accounts had been transferred. A decree was issued (2016) to allow secondary project accounts at commercial banks until December 2017. 　 In sum, the operation contributed to reducing ineligible officials from the payroll, but did not succeed in reconciling the payroll and civil service database. The operation’s support also enabled the transfer of externally funded project accounts from commercial banks to the Central Bank. Rating Substantial Page 5 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) PHREVDELTBL PHREVISEDTBL 5. Outcome Relevance of objectives is rated substantial, since objectives addressed well-identified development constraints and were aligned with country programs and the WBG's strategy. Relevance of design is rated modest, since outcomes in the results matrix reflected primarily outputs, with causal chains consisting primarily of the direct links from decisions to actual implementation of those decisions. Some important achievements were recorded, although overall attainment of objectives was incomplete. While the goal of reconciling the payroll and civil service data bases was not reached, the efficacy of the second objective is rated substantial, since the operation contributed to reducing ineligible officials from the payroll, and enabled the transfer of externally funded project accounts from commercial banks to the Central Bank. Overall, shortcomings are considered moderate, and outcome is rated moderately satisfactory. a. Outcome Rating Moderately Satisfactory 6. Rationale for Risk to Development Outcome Rating Development outcomes remain exposed to high risks from political instability (particularly during the election season scheduled for the end of 2018, chronic weaknesses in governance and capacity, and vulnerability to weather shocks. Political instability and weak governance can provide scenarios for reversal of tax expenditure reforms that removed tax expenditures. Weak capacity can prevent budget and treasury reforms from being sustained and deepened. Weather shocks can divert resources from further implementation. a. Risk to Development Outcome Rating High 7. Assessment of Bank Performance a. Quality-at-Entry Objectives addressed significant development constraints and were underpinned by analytical activities - most significantly the 2015 Systematic Country Diagnostic - that helped underpin the choice of reform areas. The macroeconomic framework was generally adequate, albeit with risks of faltering macroeconomic reforms and economic confidence, and of inadequate external financing. The operation was unlikely to have effects on poverty or the environment. Planned implementation arrangements appeared adequate, with the Government’s Economic Council. A multi-sector group responsible for achieving the objectives of the National Development Plan was to coordinate the activities of different Ministries and other entities and to Page 6 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) report on progress. 　 There were, however, shortcomings. The operation lacked a convincing theory of change, as results largely comprised outputs reflecting implementation of governmental decisions (for example, timely publication of audits and the decision to post them). M&E design had weaknesses in that indicators were not fully adequate for monitoring even the limited objectives (see Section 9a below). Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision No supervision report was filed, and there was no indication of supervision activities in the ICR. The ICR refers only to an interim status report that GOM prepared. IDA continued with a fiscal DPO program after closing, with continuity in support for some of the specific policies covered by this operation (significantly on JIRAMA). Quality of Supervision Rating Not Rated Overall Bank Performance Rating Moderately Satisfactory 8. Assessment of Borrower Performance a. Government Performance Government undertook prior actions consistent with the objectives (e.g., initial publication of JIRAMA accounts to start publication on a monthly basis). Madagascar’s Staff Monitored Program and the subsequent Extended Fund Facility with the IMF remained broadly on track. Consultations were held with stakeholders and experts on the 2015-2019 National Development Plan to which the operation was aligned (see Section 3a above). A Partnership Group Co-chaired by the Ministry of Finance and Budget and the World Bank acted as a forum for potential providers of budget support. Nevertheless, there were significant shortcomings. In some areas (for example, the updating of the civil service roster), there was insufficient funding to ensure implementation. The MFB, as a member of the Economic Council, could have taken further steps to ensure the timely publication of the Air Madagascar financial statement. Tax exemptions awarded following the decision by the Council of Ministers were not published, and this information remained undisclosed by the Prime Minister’s Office. M&E outputs were delivered with delays and insufficient frequency. Government Performance Rating Moderately Unsatisfactory Page 7 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) b. Implementing Agency Performance --- Implementing Agency Performance Rating Not Rated Overall Borrower Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Indicators were not fully adequate for the purpose of monitoring the limited objectives. For Objective 1 (i.e., stronger reporting and more relevant information), indicators (e.g., more timely audited accounts and budget execution reports) did not capture the strength of reporting (See Section 3). Indicators for Objective 2, which covered two outputs (civil service data and reconciliation of data bases) and one outcome (reductions in benefits paid to ineligible officials), reflected objectives better (e.g., wage management). Indicators were measurable and straightforward, requiring collection of simple statistics and reform implementation actions. Baselines were similarly straightforward. The operation’s M&E is embedded institutionally, as it reflects issues receiving strong stakeholder attention (e.g., JIRAMA and Air Madagascar performance). b. M&E Implementation Planned M&E data was collected and indicators measured. M&E was implemented jointly by the Economic Council and the World Bank. The Council delivered the information, which did not present difficult quality challenges, although the ICR notes that some was belated and that reporting could have been more frequent. There is no indication that ultimate beneficiaries (e.g., JIRAMA customers) were involved in the M&E process. c. M&E Utilization M&E was limited to the simple results framework that linked reforms primarily to their outputs. As the DPO was a single tranche operation, opportunities for shifts in the program were limited to the one-year implementation period. The ICR notes that the Government shared information with external partner through the Partnership Group. Ownership of M&E by stakeholders (e.g., the Government, the World Bank Group, the African Development Bank, and the IMF) appears strong and stakeholder attention is likely to be sustained. M&E Quality Rating Page 8 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) Modest 10. Other Issues a. Environmental and Social Effects The ICR did not report any effects of the operation on poverty or the environment. b. Fiduciary Compliance The ICR did not report on fiduciary compliance. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Development outcomes remain exposed to significant risks from political instability Risk to Development (particularly during the Substantial High Outcome election season at the end of this year, chronic weak governance and capacity, and weather shocks. Moderately Bank Performance Moderately Satisfactory --- Satisfactory Significant shortcomings included inadequate funding Moderately Moderately Borrower Performance for implementation in some Satisfactory Unsatisfactory areas, and tax exemptions remained unpublished. Page 9 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons IEG draws the following two lessons: 　 1. IDA impact from stand-alone DPOs will benefit from consistency with long term goals. In one area (JIRAMA, the power company), Madagascar’s first four DPOs, including that covered by this review, sequentially addressed various issues (control of leakages, statistics, procurement, and power generation contracts). Yet the long term result sought ("a well-managed company") remains elusive. A more integral approach would diagnose the various sources of losses and support those measures with significant loss reduction impacts. Timely parallel support through Investment Project Financing operations and Non-lending Technical Assistance remains critical, particularly in low capacity settings. 　 2. A program that relies on multiple government agencies and stakeholders is likely to run higher risks. This operation targeted actions by MFB, MCS, JIRAMA and three other agencies. As a result performance varied, with some agencies (e.g. MFB, the coordinating agency) achieving better results than others (e.g., MCS). Stronger leadership and incentives (e.g., providing budgetary compensation for undertaking costly reforms) may help address those risks. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides useful information. Its background section covers well Madagascar’s economic and political developments, although it could have afforded more attention to the macroeconomic and fiscal framework surrounding the operation. Its relevance section provides a useful review of the adequacy of indicators. Its review of outcomes provides additional information that complements the operation's indicators and targets. The Bank performance section could have provided more detail on the challenges faced during implementation and IDA’s role in addressing those challenges. Lessons could have been better thought through - they largely repeat findings already indicated in previous sections. Page 10 of 11 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MG - Resilience DPO(P153084) a. Quality of ICR Rating Substantial Page 11 of 11