ICRR 13018 Report Number : ICRR13018 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/08/2008 PROJ ID : P065301 Appraisal Actual Project Name : Economic US$M ): Project Costs (US$M): 45.7 35.8 Management Capacity Building Project Country : Nigeria Loan /Credit (US$M Loan/ ): US$M): 20.0 19.3 Sector Board : EP US$M ): Cofinancing (US$M): 4.8 2.0 Sector (s): Central government administration (89%) Law and justice (11%) Theme (s): Other accountability/anti-corr uption (29% - P) Public expenditure financial management and procurement (29% - P) Economic statistics modeling and forecasting (28% - P) Judicial and other dispute resolution mechanisms (14% - S) L/C Number : C3345 Board Approval Date : 05/11/2000 Partners involved : DFID, EU, USAID, Closing Date : 12/31/2005 12/30/2007 and JPN Evaluator : Panel Reviewer : Group Manager : Group : Tim L. De Vaan James Sackey James Sackey IEGCR 2. Project Objectives and Components: a. Objectives: EMCAP constituted a first step in assisting the Federal Government of Nigeria (FGN) to strengthen economic governance by: a. Fostering greater transparency and accountability in fiscal operations; b. Strengthening the systems designed to improve the efficiency of resource use, while discouraging -- via public transparency -- those elements that have deterred such efforts in the past; c. Improving the quality and timeliness of data for economic and social analyses, as well as their wide dissemination; and d. Enhancing the capacity to carry out, coordinate, and monitor macroeconomic and structural policies . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): EMCAP focused on eight components plus a ninth component covering support for the project unit : (1) Improvement of statistics: This component included assistance to the National Bureau of Statistics (NBS) to strengthen its capacity to improve the quality of the statistical base and the timeliness of the data releases, with the aim of bringing Nigeria into full compliance with the General Data Dissemination Standard (GDDS). It focused specifically on: (i) economic statistics and national accounts; (ii) CPI; (iii) poverty monitoring; (iv) social statistics; and (v) primary data collection. Cost: Appraisal US$8.51 million; Actual 7.85 million. These amounts reflect all donor funds. In the case of this component, the Bank itself did not plan any funding . (2) Improvement and monitoring of public expenditures : This focused on strengthening the prioritization, allocation, and monitoring of public expenditures by : (i) providing local and external technical assistance to the National Planning Commission (NPC) and the Budget Office, aiming to form the basis of an annual PER exercise which would be tied to the budget cycle; (ii) financing publication of summary revenue and expenditure data, and the institutional strengthening in the Federal Ministry of Finance (FMF); and (iii) strengthening the capacity of the NPC to monitor the execution of the Public Investment Program . Cost: Appraisal US$2.94 million; Actual US$2.58 million. (3) Monitoring and Accounting of fiscal operations : This component aimed at strengthening accounting and internal auditing in the Office of the Accountant General of the Federation (OAGF) and external auditing in the Office of the Auditor General of the Federation (OAuGF), and budget monitoring activities in the National Assembly, with the goal of promoting transparency and accountability in the public sector and to improve the dissemination of data on fiscal operations. Cost: Appraisal US$7.09 million; Actual US$6.93 million. (4) Petroleum Sector Review: This was to finance a review of the petroleum sector covering financial, operational, technical, legal, and contractual aspects as well as reform of key sector institutions . Cost: Appraisal US$3.53 million; Actual US$0.22 million. (5) Strengthening the procurement activities in the public sector : (i) undertake a joint review with the FGN of procurement practices of the FGN, a sample of State Governments, and a sample of parastatal enterprises; (ii) undertake a review of financial accountability systems; and (iii) finance the launching of workshops that would address procurement and financial accountability issues as well as the implementation of the recommendations of the joint FGN/Bank review. Cost: Appraisal US$2.94 million; Actual US$2.63 million. (6) Strengthening policy development capabilities : This component aimed at enhancing policy analysis capacity and improving the efficiency and transparency of policy design and coordination in the FGN . It would finance reviews/publications on a range of topics related to policy development . Also, it sought to fund the establishment of the Global Distance Learning Center . Cost: Appraisal US$8.86 million; Actual US$7.92 million. (7) Information management system: This component dealt with the design and implementation of an Integrated Financial and Economic Management System (IFEMIS) that would link key departments and provide them with comprehensive and integrated financial information . Cost: Appraisal US$2.52 million; Actual US$2.15 million. (8) Strengthening the legal and judicial systems : This aimed at providing support to the Office of the Attorney General of the Federation (OAtGF) and the Office of the Chief Justice (OCJ) in carrying out preliminary diagnostic studies on the constraints impeding the proper functioning of the legal and judicial systems . These would lead to the formulation of a legal and judicial reform program . Cost: Appraisal US$2.86 million; Actual US$2.86 million. (9) Support for the Project Unit: This component would provide training to staff at the Project Unit in the Multilateral Department of the Federal Ministry of Finance (FMF). Cost: Appraisal US$2.82 million; Actual US$2.65 million. Because of urgency, component 4 -- the Petroleum Sector Review -- was dropped from EMCAP financing in 2000 and funded out of the Bank's administrative budget . Its original EMCAP funding was allocated across other components and used to compensate for exchange rate losses . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost and Financing: The actual costs of EMCAP were considerably lower than the estimates at approval : US$35.8 million against US$45.7 million. The Bank disbursed almost all of its planned US$ 20.0 million, while counterpart funds were only about 40 percent of estimates (due to delays in budget disbursements ) and bilateral funds fell short for reasons such as delays, expiration, and continuation through a follow up project . This large Economic Reform and Governance Project (ERGP, US$179.2 million at appraisal, with US$140 million from the Bank) overlapped with EMCAP both in terms of time -- it started in 2005 -- and in terms of content, for instance in statistics and procurement. Dates: The closing date was extended by more than two years, from June 2005 to December 2007. This change was made in order to attempt to complete the construction of the Global Distance Learning Center under objective d. 3. Relevance of Objectives & Design: Relevance of Objectives: The objectives of this project were relevant at the inception of the project and continue to be relevant to date. The objectives are consistent with the Bank ’s assessment of the areas of priority assistance needs in capacity building and technical assistance identified in the 1993 and 2000 CAS. Abandonment of reform efforts in the early 1990s had left Nigeria in a state of weak economic governance and management . Financial mismanagement and corruption led to poor economic performance, and the lack of transparent and reliable economic and social data led to a loss of developmental credibility . Problems existed in basic government administration, specifically in: the national accounts, poverty statistics, accounting and auditing capabilities, public expenditure priorities and allocations, project monitoring, and the reconciliation and integration of data from key financial management agencies of the FGN . EMCAP constituted the first step in addressing these issues after a change in administration. Relevance of Design: The project was a response to government demand, was informed by relevant stakeholders, and incorporated lessons from the Economic Management TA Project (closed: 1999) and from other successful Bank operations . A 2000 CPAR was conducted as part of EMCAP -- which influenced the work program on procurement. The project was broad-ranging, especially given its relatively small financial commitments . The PAD's design in terms of objectives and components shows some flaws : there is some overlap in the 4 broad objectives, and the components were not assigned a place under one of the four objectives -- which compromised a results-based rationale. Because of this, the ICR could not assign -- ex post -- the Legal and Judicial component to a specific objective. Also, the parallel financing approach led to inconsistent donor requirements that caused delays in the first years of the project . In addition, the project lacked a sound results framework with strong performance indicators and time-bound targets to allow for sound M&E (also see section 10). 4. Achievement of Objectives (Efficacy): The achievement of each objective is analyzed below : a. Fostering greater transparency and accountability in fiscal operations . In improving monitoring and accounting of fiscal operations, some results were made on the output level . EMCAP contributed to the reorganization of the Office of the Accountant General of the Federation (OAGF) that added two departments, it helped reinforce OAGF's statutory function and its capacity to undertake internal auditing, and -- through training -- introduced greater consistency in accounting and controls in line ministries . EMCAP helped reinforce the oversight role of the Public Account Committee, and aided in reviewing the outdated 1958 Audit Act. However, EMCAP only partially met the aim of publishing candid reports from the Office of the Auditor General of the Federation (OAuGF), and there is no evidence of the planned penalties for violators . Also, corruption surveys were planned but only one was conducted, and the results were not published . Moreover, the aim of an operational IFEMIS was not met, even though the overlapping 2004 ERGP also supports this topic. IFEMIS infrastructure is in place, but the publication through the system of the budget expenditures and receipts has not taken place as the software has not been installed . Achievement: negligible b. Strengthening the systems designed to improve the efficiency of resource use, while discouraging —via public transparency—those elements that have deterred such efforts in the past . EMCAP supported the ability to carry out PERs by training staff of the National Planning Commission (NPC) and the Budget Office (BO), and by reorganizing responsibilities . This resulted in 4 sectoral PERs, although this still falls short of the eventual goal of having a annual PER . In addition, NPC staff were trained in the supervision of capital projects, although there is no evidence that this led to the desired output of institutionalizing a solid economic justification for capital projects . EMCAP and the PERs seemed to have influenced the improved allocation of resources, e.g. more funds are being channeled to key MDG ministries . Under EMCAP, a joint FGN/World Bank CPAR was undertaken in 2000. This led to the adoption of the CPAR recommendations by the Executive Council, to a revision of procurement practices and the introduction of new procedures, and to the approval of a Public Procurement Bill . A procurement manual was developed, and value-for-money audits were carried out for 60 recurrent and 60 capital projects. Work on this topic is continued under the 2004 ERGP. Achievement: substantial c. Improving the quality and timeliness of data for economic and social analyses, as well as their wide dissemination. The Bank played a critical role in coordinating, and providing technical assistance to the production and publication of national social data and poverty profile . EMCAP provided training to 360 National Bureau of Statistics staff, enhanced data production through the introduction of digital technology, and strengthened IT capacity . This resulted in improved and quarterly available GDP data, newly calculated monthly CPI data, newly published indicators of social statistics, and 2004 and 2006 poverty surveys. These should eventually pave the way to compliance with the General Data Dissemination Standard . Achievement: substantial d. Enhancing the capacity to carry out, coordinate, and monitor macroeconomic and structural policies . EMCAP supported the creation of the Economic Policy Coordination Committee (EPCC), and provided training and workshops aimed at improving economic policy coordination and capacity building efforts at the EPCC and similar institutions. The ICR refers to a 2005 survey that indicates positive feedback on EMCAP training, but it does not cite specific results related to capacity in macroeconomic and structural policies . Policy studies were undertaken for about half of the topics planned in the PAD : fiscal decentralization, trade policy, private sector development (several), and the pension system. Lastly, EMCAP did not meet its goal of creating a Global Distance Learning Center, even after the project was extended by more than two years to allow for its establishment . Achievement: modest Finally, as mentioned above, the ICR discusses the component on the Legal and Judicial system outside the project's 4 objectives. The ICR lists few concrete results : EMCAP financed a review of the judicial system, held 6 zonal stakeholder workshops, and helped stock the library of the Supreme Court and the Judicial Council . However, there is no evidence that EMCAP met its aim of helping to formulate a legal and judicial reform program . 5. Efficiency (not applicable to DPLs): No measures of efficiency (ERR/FRR/NPV) were calculated. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re-estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The outcome rated for the project is moderately satisfactory . EMCAP's results -- although fewer than set out in the PAD -- constitute important first steps towards improving economic governance . This way, EMCAP was able to gain momentum on governance reform and pave the way for the much larger follow -up ERGP. Successful areas were the efficiency of public resourse use and the quality and timeliness of data that received an important boost through EMCAP. The project generated some results on the output level in the areas of transparancy and accountability in fiscal operations, macroeconomic and structural policies and legal and judicial reform --although with some gaps. In particular, the IFEMIS is not operational, and the Global Distance Learning Center has not been created . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated moderate. Progress such as the timely publishing of economic and social data, and the adaptation of good practices in procurement are likely to be sustainable . One of the main threats to sustainability -- not discussed in the ICR -- could be the high oil prices, as experience in the 1980s and 1990s has shown a negative correlation between high oil revenues and political commitment to economic governance reforms in Nigeria. However, the fact that EMCAP was followed up by ERGP reflects the likelihood that commitment will be sustained in the medium term. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at Entry: The Bank’s quality at entry is rated as : moderately unsatisfactory , which reflects a more critical stance than the 2001 QAG quality at entry assessment (satisfactory). EMCAP was the Bank's response to client demand, was informed by relevant stakeholders, and integrated lessons from EMTAP and other Bank projects. However, the project components and objectives did not reflect a clear results chain, and the project design lacked a robust M&E framework . Also, parallel donor financing proved to be a complicating factor, leading to some delays in project execution . Quality of Supervision: The Bank’s quality of supervision is rated as : moderately satisfactory . Regular supervision was carried out by the TTL based in the field, who was changed once during the implementation period. Also, the ICR mentions that the Project Unit and Project Executing Agencies felt that the supervision team was proactive and supportive . However, some ISRs had limited information content, and the ICR mentions that many ISRs were filed without management review . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government Performance: After a period of standstill in economic governance reforms in the 1990s, FGN's new administration showed strong initiative and commitment, and was directly involved in EMCAP preparation (including content). Commitment waned somewhat after the startup in 2000, which was reflected in the following : counterpart funds sometimes had considerable delays, the Steering Committee at the Ministry of Finance responsible for e.g. overseeing project implementation failed to be effective, and the small Japanese grant to be used for IFEMIS lapsed due to non -utilization. Also, the FGN was late to deliver land to be used for the Global Distance Learning Center, which it attributes to a communication gap with the Bank . Implementing Agency Performance: As set out in the PAD, the Project Unit (PU), located in the Ministry of Finance, was to monitor and coordinate project implementation across 9 Project Executing Agencies (PEAs). Parallel financing requirements proved demanding to the PU, as capacity was already low . Donor coordination issues improved after the MTR. Around that time, office equipment deficiencies were also addressed . However, the PU did not produce any M&E reports . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: The M&E framework of the project was very weak . The PAD identified 4 key performance indicators and 6 outcome/impact indicators, although these were really output indicators or intermediate outcome indicators at best . They were not quantified and not properly linked to project objectives or components, and hence, did not constitute a robust results-based framework usable for M&E. Because of this, and for reasons of low capacity, the Project Unit did not produce the planned quarterly M&E reports . The Bank undertook semi-annual supervision missions and a 2002 MTR. Baseline and final targets were identified late in the process (2005), but these were subject to change and failed to fully cover all project objectives and components alike . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The ICR does not raise any safeguard or fiduciary issues . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately EMCAP lacked a robust M&E Satisfactory Unsatisfactory framework with components linked to objectives. Many ISRs were filed without management review. See sections 8 and 10. Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: These lessons are in broad alignment with those mentioned in the ICR : 1. The absence of a robust results -based framework with measurable (and quantified) indicators linked to project objectives and components will impair M&E and complicate findings on the achievement of outcomes . 2. Lack of harmonization of donor procedures regarding disbursement requirements will cause delays and unduly burden the implementing agency. The latter will be able to deliver only when it is sufficiently staffed and equipped . An effective steering committee is needed to complement the project unit . 3. The effects of training can only be observed when measured against indicators of success . In the absence of these, the measurement of increased capacity is problematic . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the ICR is satisfactory, but marginally so . It provides a candid account of the activities undertaken by under EMCAP, but is hampered by the absence of a results -framework that allows for systematic tracking of relevant indicators and targets. Because project objectives and components are not clearly linked in the PAD, the component on the legal and judicial system is discussed outside of the 4 project objectives in the ICR, and hence, is not included in its ratings. Given these limitations, the ICR still lacks specificity . The ICR should have better distinguished between results attributable to either EMCAP or ERGP . Also, a table detailing the planned and actual costs of each component, broken down by donor (and FGN), would have been welcome. Finally, a more specific account of the EMCAP feedback survey could have been provided . a.Quality of ICR Rating : Satisfactory