ICRR 13521 Report Number : ICRR13521 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/23/2011 PROJ ID : P074448 Appraisal Actual Project Name : Governance And US$M ): Project Costs (US$M): 58.0 42.9 Institutional Development Project Country : Madagascar Loan/ US$M): Loan /Credit (US$M): 30.0 38.2 Sector Board : PS Cofinancing (US$M): US$M ): - - Sector (s): Central government administration (73%) Vocational training (27%) Theme (s): Other accountability/anti-corr uption (33% - P) Public expenditure financial management and procurement (33% - P) Administrative and civil service reform (17% - S) Education for the knowledge economy (17% - S) L/C Number : C3829 Board Approval Date : 11/18/2003 Partners involved : Closing Date : 06/30/2009 06/30/2009 Evaluator : Panel Reviewer : Group Manager : Group : Rene I. Vandendries Navin Girishankar Ismail Arslan IEGCC 2. Project Objectives and Components: a. Objectives: The initial project development objective was to assist the government of Madagascar in implementing the poverty reduction strategy which is set out in the Poverty Reduction Strategy Paper (PRSP) by (i) rationalizing and modernizing budget and public expenditure management, (ii) improving accountability and transparency of government operations and, (iii) strengthening the capacity of public institutions to deal with complex change processes. In 2006, the government dissatisfied with the PRSP, which it perceived as an external product, replaced it with its own Madagascar Action Plan (MAP) which led to an official restructuring of the project in mid -2007. The new project development objective was defined as improving the efficiency and transparency of government and public services in Madagascar in line with the MAP . In essence, the modification in the development objective was minor, essentially to align it better to the government's new strategy and add emphasis on statistical data generation and monitoring and evaluation . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 05/22/2007 c. Components (or Key Conditions in the case of DPLs, as appropriate): The initial project had three components : 1. Improvement of transparency and economic governance through support for comprehensive reform of the public finance system. 2. Capacity building and strengthening of local training institutions to promote institutional development . 3. Creation of a project management system . At restructuring, these three components were reorganized into four components, while an extra component was added: monitoring and evaluation of the MAP and key statistical surveys . At that time, the original project funds were largely disbursed or already committed . The additional financing requested of US$ 5 million was meant to cover the extra component. As a result, the project now had the following five components, achievements against which are evaluated below without the need to distinguish between "original" and "revised" project. 1. Reform of the public finance system . 2. Capacity building and public sector service delivery . 3. Rule of law and fight against corruption 4. Monitoring and evaluation of the MAP and key statistical surveys . 5. Program coordination. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was approved on November 18, 2003 for US$ 30 million equivalent, with a closing date of June 30, 2009. The project was restructured on May 22, 2007 when additional financing of US$5 million equivalent was approved. The project closed on schedule on June 30, 2009. The difference between the appraisal amount and actual disbursement is due to the US$ 5 million additional financing and to exchange rate changes between the SDR and US dollar. The borrower contribution, planned at US$ 28 million at appraisal, was estimated at US$ 4.7 million at closing. A political crisis at the end of January 2009 led to a "de facto government" and the Bank's OP 7.30--Dealings with De Facto Governments--has been in effect since. There has been no Bank funding to Madagascar since and only limited AAA. As a result also, the ICR was prepared without the necessary contacts with government officials. 3. Relevance of Objectives & Design: In view of the challenges facing Madagascar's public sector, the development objectives of the project were relevant at appraisal as well as at restructuring, when added emphasis was placed on data gathering and monitoring and evaluation. They were also consistent with the Bank's CAS . The development objectives remain relevant today even though the de facto government's development strategy is unclear at this stage . The design of the project was well aligned with the government's PRSP /MAP and the different components complemented each other well. However, there were important shortcomings in the design of the operation . First, para 3 of the ICR points to the "limited contribution of decentralized structures to service -delivery" as a major issue; yet the project did not include reforms to strengthen local government capacities to tax and spend nor to reforms of inter-governmental relations. In addition, unlike the experience in several other African countries where reform ownership and capacity is built by giving government institutions at all levels access to capacity building resources in return for tangible improvements in performance, no such process was envisaged in this operation. Instead, the project relied heavily on direct Presidential oversight and on transitory technical advisors for its implementation. 4. Achievement of Objectives (Efficacy): Component 1: Reform of the public finance system . The objective was to achieve substantial improvements in public financial management in a variety of areas . Much was achieved. A modern computerized integrated financial management system was introduced (SIGFP) thereby greatly facilitating budget preparation and execution . On the revenue side, policy and administrative reforms included consolidation of tax and customs reform and restructuring and modernization of the management of the Tax Directorate and the large taxpayer directorate to allow for more transparency on tax declaration and collection. On the other hand, a new procurement code was put in place in 2004, following international standards, but compliance is limited . Also, internal and external oversight mechanisms remain ineffective. At the same time, the ICR provides no detail on the relative contribution of the Bank versus other donors, such as the IMF, who are likely to also have focused on this component . On balance, IEG rates efficacy for this component substantial . Component 2: Capacity building and public sector service delivery . The objective was to strengthen the institutions at the center of the reform process through training and capacity building, and thereby improve service delivery . Results indicators included such items as the output of training courses, the percentage of people satisfied with delivery of selected public services, or the percentage of ministries meeting the targets of their annual work programs . With the exception of the latter, where the result fell short of the objective, none of the outputs were measured . In addition, the 2009 political crisis undid much of the progress that may have been made as individuals left the public service and some institutions were abandoned. Efficacy for this component is rated negligible. Component 3: Rule of law and fight against corruption . The objective of this component was to promote transparency, accountability and good governance, including reducing corruption. The Bank helped the government introduce anti -corruption legislation and helped set up anti-corruption institutions. Most targets set, such as e .g. the number of courts that operate in line with agreed service standards, a reduction in the proportion of average expenditure paid by households for bribery for public services, or an improved rating for the WBI Rule of Law indicator, were not met . Madagascar's rank in the Transparency International Corruption Perception Index, an important outcome indicator for the project, fluctuated over time but did not improve . Efficacy for this component is rated negligible. Component 4: Monitoring and evaluation of the MAP and key statistical surveys . The objective was to improve the development effectiveness of the project by providing timely information on progress. A monitoring and evaluation system was set up but information on achievements and progress was not collected. The target for cartographic surveys was exceeded . Efficacy for this component is rated modest. Component 5: Program coordination. The objective was to support project implementation . The intermediate indicators, annual progress reports submitted on time and completion of annual work plans, were achieved . But, while the location of the PIU in the President's office had the advantage of benefiting from the President's personal interest, it also made project execution difficult at times because of regular unplanned requests for support from the Presidency which complicated the coordination between the PIU and government institutions . Efficacy for this component is rated modest. 5. Efficiency (not applicable to DPLs): N.A. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project was and remains relevant . However, apart from some significant progress in reforming the public finance system, achievements in other areas, such as capacity building, public service delivery, corruption control, or monitoring and evaluation, have been modest to negligible . On balance, the outcome was moderately unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Risks were identified as substantial from the outset because of the complex changes supported by the project and the major capacity building needs . However, among others, a monitoring and evaluation system was set up to help mitigate some of these risks but it proved ineffective . Furthermore, the 2009 political crisis and the emergence of a de facto government cast doubts on the government's commitment to reform . a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: In close collaboration with the government, the Bank designed a project well suited to address the challenges facing the performance of Madagascar's public sector in several areas . The project complemented other Bank operations, including the PRSC and sector operations . The Bank was also flexible to adjust to new government priorities . But, progress on project implementation was not well documented in the ISRs. This contributed to an inadequate appreciation of the limited progress being made, to a lack of corrective action and to the ultimate less than satisfactory outcome . Also, as discussed in Section 3 above, there were important shortcomings in the design of the operation . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: At the time of project appraisal the government was fully committed to the project and the proposed reforms . Following the 2009 political crisis government commitment to reform is essentially unknown . The centralized decision-making affecting the PIU led to implementation delays and friction between the PIU and the government institutions meant to benefit from the support of the project . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: At the time of project restructuring an adequate M&E framework was designed , linking project activities to expected outcomes. However, some of the indicators were poorly chosen . The Transparency International Corruption Perception Index e.g. is too broad to assess Bank contribution and is also of limited technical value . The M&E framework was not implemented, however. Baseline data were not developed and outcomes were not monitored. However, under the project, a citizens' report card on public service delivery in health was developed, which was used and incorporated in the follow-up second Governance and Institutional Development Project. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): N.A. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development High High Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Two of the lessons in the ICR stand out . First, leadership support at the executive level of government has the advantage of prompt action on reform but may also undermine support from other levels of government, thereby jeopardizing the long term sustainability of the reform process . Second, regular and better monitoring and reporting on progress made towards objectives are vital to improve the effectiveness of ongoing and future assistance . 14. Assessment Recommended? Yes No Why? Upon closing of the follow-up project (Governance and Institutional Development Project No . 2) an assessment of the two projects would be valuable especially to evaluate the pursuit of public sector reform in a volatile political environment. 15. Comments on Quality of ICR: The ICR clearly suffers from the fact that it was not possible to get official feedback from the government . The only interaction was with a few technical staff in the PIU and some other public institutions . Under these circumstances a credible effort is made to review the project, its implementation, and obstacles encountered . Enough evidence is presented to make a sensible evaluation of the project's outcome . a.Quality of ICR Rating : Satisfactory