Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) Report Number : ICRR0021058 1. Project Data Project ID Project Name P112821 ML-Governance and Budget Decent. TA Proj Country Practice Area(Lead) Mali Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-49140 15-Dec-2014 12,000,000.00 Bank Approval Date Closing Date (Actual) 28-Apr-2011 30-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 12,000,000.00 0.00 Revised Commitment 12,000,000.00 0.00 Actual 9,995,117.02 0.00 Prepared by Reviewed by ICR Review Coordinator Group Jorge Garcia-Garcia Judyth L. Twigg Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components a. Objectives The project appraisal document (PAD) and the financing agreement (FA) define the development objective as “to strengthen capacity for public financial management at decentralized level for improved budgetary decision-making and enhanced transparency and oversight” (PAD, page 7; FA, p. 5). The project’s primary targets were civil servants, elected officials, and representatives of civil society. Four regions were selected as primary beneficiaries of the health and education activities the project financed. To examine efficacy in section 4 the review breaks the objective into four parts: Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) • strengthen capacity for public financial management (PFM) at decentralized level (objective 1) • improved budgetary decision-making (objective 2) • enhanced transparency (objective 3) • enhanced oversight (objective 4) Each of the four sub-objectives is related to components 1-3 reviewed in section 2d below: objective 1 with component 1; objective 2 with component 2; objectives 3 and 4 with component 3. The objectives remained unchanged, but the values for the indicators of outcome, output, and related targets were revised upward in two restructurings of the project. Most of the targets were met; therefore, a split rating is not required. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project consisted of four components. Components 1-3 financed the cost of consultants, the purchase of computer hardware, computer software, other equipment, solar power kits, scholarships, travel costs, printing and copying services, and expenses for organizing workshops and training events. Component four financed the costs of implementing and coordinating the project. Component 1: Strengthening capacity for PFM at decentralized level (US$7.1 million at appraisal, US$ 6.7 million at closing). The component financed expenses for (a) formulating a local taxation framework and strengthening capacity for implementing it; (b) strengthening capacity to improve budgetary decision-making at decentralized levels; (c) strengthening the regulatory framework for procurement and the capacity to implement it; and (d) enhancing accounting and internal controls at deconcentrated levels. These activities were carried out under four subcomponents: (a) resource mobilization at local level (US$1.6 million); (b) budget preparation and monitoring (US$1.4 million); (c) procurement (US$2.4 million); (d) accounting and internal controls (US$1.7 million). Component 2: Piloting new public financial management tools in the education (MEALN) and health (MS) sectors (US$2.0 million at appraisal, US$1.92 million at closing). The component financed expenses related to preparing and piloting criteria for allocating resources to the education and health Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) sectors to territorial collectivities. The activities financed were carried out under two subcomponents: (a) education pilot (US$1.2 million); (b) health pilot (US$0.8 million). Component 3. External Oversight and Community Participation (US$1.7 million at appraisal, US$2.03 million at closing). The component financed expenses related to (a) technical assistance for building capacity for fiduciary oversight at the centralized and decentralized levels; the activities included, among others, training new auditors and drafting a code of ethics for financial judges; (b) establishing a framework for the participation of stakeholders in monitoring and evaluating the allocation and use of public resources; and (c) establishing and implementing mechanisms for collecting and treating financial data to inform community surveillance committees. These activities were carried out under two subcomponents: (a) external audit (US$1.3 million); (b) participation (US$0.4 million). Component 4. Project Implementation and Coordination Unit (US $ 1.2 million at appraisal, US$1.35 million at closing). The component financed the costs of executing, monitoring, and evaluating the project. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs at appraisal were US$12 million. Contingent costs were not calculated. The credit financed all project costs. The ICR reports actual project costs of US$12 million at closing (p. 56) and actual disbursements of US$10 million (p. 2) but does not explain why the values differ. The ICR reports in another section (page 20, last bullet of par. 1) that the amount undisbursed was $1.2 million, implying that total disbursement and project costs were $10.8 million. The ICR reports that more project activities were executed with a smaller budget. The ICR does not report on costs financed by the borrower such as salaries of public servants and other expenses related to the project. On April 28, 2011 the Board approved the project, which became effective on August 8, 2011. The project’s closing date of December 15, 2014 was extended twice, to June 30th, 2016 and to June 30th, 2017, when it closed. The project was restructured twice, June 27, 2014 and May 6, 2016. The restructuring was done using a level 2 approval process both times. Delays in completing critical activities under the initial schedule forced the first restructuring; delays in procurement processes forced the second restructuring. With the two extensions came revisions in the results framework, changes in some activities, and reallocation of some funds to “ensure alignment with government priorities” (Project papers, Restructuring 1, p. iv, and Restructuring 2, p. 8). 3. Relevance of Objectives Rationale Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) The project selected relevant objectives. They were aligned with the priorities of the Country Assistance Strategy Progress Report (CASPR) of FY08-11, of the Interim Strategy Note (ISN) of FY14-15, and of the Country Partnership Framework of FY16-19. The three programs sought to improve public sector performance under the headings of strengthening public sector performance (CASPR strategic objective 2), laying the foundation for long-term accountability and stability (ISN, Priority Area I), and improve governance (CPF, focus area 1). The objectives broke down into subobjectives that sought to improve public resource management at the central and local levels, decentralization, better budget control, more transparent budget practices, better procurement practices, and better link between budget and sector strategies. The strategies were relevant to the challenges of the country and were aligned with the government’s Second Growth and Poverty Reduction Strategy Framework (GPRSF) for 2007-2011, with the Third Growth and Poverty Reduction Strategy Paper (GPRSP-3), and action plans prepared before the new government took over in September 2013. The project objectives were aligned with the World Bank Group strategies for Mali and with the government’s development plans. The operation continued the Bank’s support in public financial management, which it provided through analytical work, policy dialogue, and development policy loans. The operation supported the action plan to strengthen and modernize public finance management (with decentralization). It also tried to address some bottlenecks identified in the public expenditure management and financial accountability review of 2008, in the action plan to strengthen and modernize public financial management of 2010, and in the World Bank’s Systematic Country Diagnostic for Mali (2015). The operation also sought to improve problems of governance and public financial management. Mali ranks low in the Worldwide Governance Indicators, below the 20th percentile in government effectiveness, and in the Open Budget Initiative score, whose number (39/100 in 2017) indicates there is minimal information on the budget. The project sought to help Mali ameliorate a serious problem of governance by improving transparency and oversight, improve decisions on budget and strengthen its capacity for managing public finances, advance the government’s decentralization policy, and build local administrative capacity. Rating High 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Objective 1: Strengthen capacity for public financial management at decentralized level Rationale Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) To achieve the objective the technical assistance activities aimed at improving (a) resource mobilization at the local level, (b) budget preparation and monitoring, (c) procurement, and (d) accounting and internal controls. The expected outcomes were to reduce the time required to treat procurement files and the time needed for the central government to consolidate data on fiscal revenues at the level of territorial collectivities (collectivité territorial (CT), lower levels of government). The targets were achieved. On procurement, the time to treat files fell to 67 days, below the revised and original targets of 73 and 80 days and the baseline value of 98 days. On consolidating data, it took one month to consolidate data on fiscal revenues at the local level, below the three months target and the 18 months baseline value. Intermediate results • Resource mobilization at the local level • IT equipment and solar kits were installed in 69 treasury representations, fiscal centers, and financial controllers at the territorial collectivity level. The number exceeds the target of 67. • A guide on fiscal administration was produced. Target met. • Budget preparation and monitoring. The project financed building capacity to improve budgetary institutions and the medium-term expenditure framework (MTEF). The training covered key public financial management functions and was received by 6,270 people, of whom 976 were women, in central and local government and in civil society organizations; the numbers exceed the targets of 6,000 and 800. Some areas of public financial management improved between 2010 and 2016, while the quality of the overall system seems to have remained unchanged. The PEFA indicators for effectiveness in collecting tax payments (PI-15) and effectiveness of internal controls for non-salary expenditure increased from C+ to B+ ; other relevant indicators remained unchanged: timeliness and regularity of accounts reconciliation (PI-20) and orderliness and participation in the budget process (PI-11). The indicator for multi-year perspective in fiscal planning, expenditure policy and budgeting (PI-12) fell from A to B+, suggesting that the support for MTEF is not yet reflected in long-term planning. • Procurement • The Mali regulatory framework was put in line with regional guidelines. • The government produced and disseminated a procurement buyers guide, local-level procurement manuals, and a dossier of key procurement regulations. • SIGMAP, an information technology (IT) procurement system, was installed. It facilitates procurement processes at the central level and connections to the regions covered by the project. • Oracle software was installed. It allows the Department of Finance and Materials (DFM-credit managers) to connect with the procurement departments at the central and local levels in the targeted regions. Over the course of the project, 60 interconnections were made between the DFM, the General Procurement Department, and the Delegation of Public Services, exceeding the target of 48 connections. • A procurement archiving system was installed. The national level procurement office uses it to organize and manage procurement documents. • The intermediate results indicator associated with this activity was to post a procurement buyer’s guide in the website of the Ministry of Economy and Finance. The indicator was met. • Accounting and internal controls. The project financed the installation of IT and solar equipment for Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) local treasury offices and the deployment of a software package (COLLOC) for consolidating the data on revenues collected at the local level in the four pilot regions. In total, 76 local treasury offices have been equipped with estate management and COLLOC software, exceeding the target of 70 offices. These actions have reduced the time it takes to consolidate accounts and led to adopting a harmonized template for the management account (Comptes de Gestion). Rating High PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Objective 2: Improve budgetary decision making Rationale To achieve this objective the technical assistance supported activities in which public financial management tools were applied as pilots in the education and health sectors. The pilot projects supported (a) preparing and implementing criteria to allocate resources to local governments and service providers in selected regions; (b) preparing guidelines for supervising education-related projects and managing public and community health centers; and (c) introducing beneficiary surveys and surveys to track public expenditure and use them as management tools. The expected results were quarterly reports informing about the resources transferred to the education sector in the selected regions and their use of the resources. The result was achieved: four reports were produced in 2016 and two reports were produced in 2017 covering the last two quarters of 2016. The results framework for the project did not contemplate preparing a report for the expenditures on the health sector. While the formal results were achieved, it is not possible to say that decision making on the budget at the central, local and sectoral levels improved. The ICR informs that between 2013 and 2015 the discrepancy between actual and budgeted expenditures fell, and the allocations to education increased in real terms. While this information points to improvement in some areas, the ICR does not provide information to assess what happened in the clusters associated with budget decision making: revenue, budget process, expenditure, balancing, and budget implementation. The lack of this information has to do with shortcomings in project design (e.g., no definition of what budget decision making means and, given that, what problem the project sought to solve), not with shortcomings in the ICR [https://www.coursehero.com/file/10417786/explaining-budgetary-decision-making-in-terms-of-the- five-decision-clusters/ for the categories mentioned]. Intermediate results • Allocation criteria. A report was produced on the criteria for transferring resources for education to territorial collectivities. The target was met. • Equipment. Sixty-two territorial collectivities were equipped with solar kits and IT hardware and Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) software. That number exceeded the target, 10. • Surveys. The beneficiary and public expenditure tracking surveys were carried out. The target was met. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Objective 3: Enhance transparency Rationale To achieve this objective the technical assistance supported activities to increase citizen participation. The expected result was to produce citizens’ control reports to improve public service quality. The Surveillance Committee and the National Council for Civil Society produced six citizens’ control reports, one by each of the five targeted districts and one at the national level. To support its conclusion that transparency improved, the ICR reports on two indicators not included in the result framework. The first one, the score of the Open Budget Index, measures the transparency of budgets. The ICR reports scores of 35, 43, 46 for 2010, 2012, 2015, but the score for 2017, not reported in the ICR, fell to 39. The scores show a better situation in 2017 than in 2010. Even so, with a score of 39 Mali falls into the category of country with minimal budget information, down from the 2015 category of country with limited information. The score for public oversight, one of two categories reported, was 6 over 100 in 2017, which means Mali provides few opportunities for the public to be involved in the budget process; the score is higher than the 4 over 100 for 2015. The second one is voice and accountability from the Worldwide Governance Indicators. For the ICR, voice and accountability is said to have improved because Mali’s score exceeds the average score for the Sub-Saharan African region. That improvement can happen if the score for other countries deteriorates. Mali’s score for voice and accountability averaged ‑0.19 for the period 2011-2016, a poor situation. The score in 2011 was positive, 0.11, fell to ‑0.50 in 2012, rose to -0.16 in 2014, and fell to - 0.18 in 2016. These numbers indicate the situation is worse in 2016 than in 2011, when the project started. The worsening civil disorder in Mali has deeper roots than the problems the project sought to solve. Therefore, using voice and accountability is inappropriate to measure the project’s success or its shortcomings. In summary, the indicators show that voice and accountability deteriorated and information on the budget, though still minimal, improved. Intermediate results • Citizens’ budget at the local level. The target of producing at least 50 budgets during the project’s life was exceeded, with 69 budgets produced [page 18 of ICR] or 62 budgets produced [page 39 of ICR]. The ICR does not present information to assess what happened with service quality. Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Objective 4: Enhance oversight Rationale To achieve this objective the technical assistance supported activities to (a) strengthen the capacity for external audit at centralized and decentralized levels, and (b) prepare for the creation of new audit institutions. The expected results were to have the Audit Section of the Supreme Court review and judge an increasing share of the management accounts submitted by territorial collectivities. By June 30th, 2017, the Audit Section of the Supreme Court had audited 17 percent of the total accounts, below the 30 percent target. Intermediate results • Reports of the Audit Section of the Supreme Court. Two indicators were defined. First, at least one annual report is published during each of the last three years of the project. The ICR informs one report was published in 2016 but does not say whether reports were published in 2015 and 2014. Second, 11 targeted audit reports were published; the target was met. In summary, lacking further information on the annual audits, the intermediate results were partially met. • Public restitutions of the management account. The target was not met. Thirty (30) territorial collectivities undertook a public restitution of the management account and of the implementation of the social and economic development plan. The target was 50 restitutions. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale The project objective was to strengthen capacity for public financial management at decentralized level for improved budgetary decision-making and enhanced transparency and oversight. Strengthened capacity for public financial management at decentralized level was highly achieved; enhancing transparency was substantially achieved; and improving budgetary decision-making and enhancing oversight were modestly achieved. Taken together, these ratings indicate that overall efficacy was Substantial. Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) Overall Efficacy Rating Substantial 5. Efficiency To judge project efficiency it is necessary to know project costs. The ICR’s information on costs fluctuates between $10 million and $12 million (see section 2e). Neither the PAD nor the ICR inform on the local costs the project was expected to incur or incurred, and whether IDA’s financing paid fully for these costs. Lacking better information, this review assumes that project costs are equal to IDA’s financing, a sum that varies between 10 and 12 million dollars, but most likely on the low side. Since the outputs from most activities carried out exceeded the plan, it is fair to say that the cost per unit of output produced was lower than the per unit cost of planned output. With a total cost of $10 million, it could be argued that costs were 17 percent below plan if the same output had been delivered. Since most of the outputs were exceeded by between 3 and 10 percent in objective 1, which accounted for about 70 percent of project costs (ex-ante), and were mostly met in objectives 2 and 3, it is fair to say that project outputs were exceeded by 2-7 percent. Therefore, it could be argued that unit costs were 20 percent lower than expected, a significant gain. These calculations do not substitute for a proper analysis of costs and benefits. As the borrower and other stakeholders comment on the ICR, and this review agrees, “while there are gains in terms of reduced delay in the processing of procurements, it will be important to understand how the procurement reforms have resulted in improved efficiency and overall cost savings” (ICR, p. 60). In summary, while costs per unit of output fell, there is insufficient evidence on how the results benefit the country, and on whether costs could have been even lower had project execution started as planned. The ICR assesses the efficiency of the project in two ways (Annex 4). First, it assesses the extra revenues collected by the local governments as part of the project by comparing actual revenues with revenues expanded by the growth rate of GDP from their baseline values. Without further information on the relation between GDP growth and local revenue in targeted communities, there is no basis to use GDP growth to extrapolate the revenue collected without the project. The proper test would have been to compare the revenues of local governments targeted by the project with those of local governments not targeted by the project. This comparison was not done because information on local revenues of non-target localities was not collected. Second, the ICR compares the performance of revenues at the national level with the performance of local revenues in the targeted communities. The ICR concludes that because revenues at the national level fell between 2013 and 2016 and local revenues of targeted communities increased, the increase should be associated with the project. While these attempts at imputing results to the project are commendable, it is difficult to conclude from them that the results can be attributed to the project. The ICR could have calculated the efficiency of the project in an alternative manner. It could have compared the ex-ante and ex-post unit cost of delivering each of the different outputs of the project. This information should have been readily available. The ex-ante costs could have been derived from the information gathered to build the ex-ante costs in the PAD. The ex-post costs could have been obtained from the disbursements associated with each output. Then, it could have compared the two values for each product. None of this was done. With insufficient information and analysis, efficiency is rated Modest. Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project had high relevance of objectives, substantial efficacy, and modest efficiency. Therefore, the project is assessed to have experienced moderate shortcomings, and the outcome is rated Moderately Satisfactory. On public financial management, the project helped train more than 6,000 government employees and people in civil society organizations (of them, about 1000 women) and to put in place the IT infrastructure that permitted a faster handling of procurement and consolidation of data. On procurement, it takes the government 67 days to treat files, less than the 98 days it took on December 2010; on consolidating fiscal revenues at the local level, it now takes the central government one month to do so, far below the 18 months of 2014. On budgetary decision making, some results suggest that budget management has improved. For example, the allocation to education has increased in real terms over the period and the discrepancy between budgeted and actual expenditures fell from 4.4 percent in 2013 to 2.0 percent in 2015. The lower discrepancy reflects the general improvement in budget management, where the discrepancies between budgeted and actual expenditures fell from 26 percent in 2013 to 17 percent in 2015. The information on the transfers for education to selected regions improved; now, quarterly reports inform on the amounts transferred to the regions and how they use the transfers. On transparency and oversight, the Audit Section of the Supreme Court audited and judged a large share of the management accounts the territorial collectivities submitted. That effort faltered in 2017; by December 31st, 2017 the Audit Section had judged 37 percent of the accounts. a. Outcome Rating Moderately Satisfactory Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) 7. Risk to Development Outcome The risks to development outcome arise significantly from political risks and the security situation in the country. While some progress has been made in implementing the Algiers Agreement, the continuing terrorist threat and drug trafficking have made rising insecurity a significant risk to the country’s economic and political stability. These events have led to delays in the elections for president from April 2018 to July 29th, 2018, signaling the fragility of the political situation. Since the Algiers Agreement was signed in June 2015, indicators of risk for Mali show that the risk of civil war has diminished but the risks of civil disorder and government instability increased. The low capacity of the bureaucracy makes it more difficult to manage these risks and implement government programs if security and political problems rise. Although the project succeeded in building infrastructure and the capacity to manage public finance at the decentralized level, localities outside the capital are more likely to suffer more from intensified conflict and insecurity, thereby putting in jeopardy the achievements of the project, among them the institutional strengthening documented in section 4. 8. Assessment of Bank Performance a. Quality-at-Entry The project sought to solve problems present in budgetary processes, financial management, and transparency and accountability. The Bank gained knowledge of these problems through its analytical work and the implementation of projects in the public financial management area and used that knowledge to design the project. The PAD, however, did not have a good discussion of budgeting, budgetary processes, and what to expect from the reforms. Had the PAD paid attention to what it meant by improved budget decision making it would have had to realize that “meaningful budgets ... depend on trust and their ability to be calculated," and “poor countries have little to spend because they are poor, and they find it hard to spend wisely because they [budgets] are uncertain” [as noted by Wildavsky in Budgeting: A Comparative Theory of Budgetary Processes, 1986, pp. 13, 17]. The PAD did not have a clear theory of decision making behind the framework for evaluating the results of the program. The design also included monitoring and evaluation arrangements that were appropriate ex-ante but turned out to be inadequate after project implementation started. In its implementation arrangements the Bank selected a complex design, whereby a technical coordination unit was set up in the Ministry of Finance and several ministries and organizations carried out the daily activities required to execute the project. Moreover, the arrangements used Bank and government systems for procurement and financial management, making project implementation more complex than usual; this problem was compounded by the low capacity of the civil service in Mali. In timing the activities the Bank overlooked the country’s capacity constraints and the complex implementation arrangements, especially on procurement, making the time span for implementing the project too short. This shortcoming was exacerbated by the deterioration in the security situation, a problem that could not be foreseen when the project was designed. Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision For about three years, project implementation and project supervision were hampered by the coup d’état of March 2012 and the insecurity it triggered, even after the constitutional order was reestablished in 2013. The implementation arrangements noted under quality at entry and the low capacity of the implementing agencies compounded these difficulties. These problems prevented appropriate collection and consolidation of data in the project coordinating unit at the Ministry of Finance. Despite the difficulties the Bank supervised the project regularly, learning about its achievements and shortcomings and informing its decision to restructure the project, reallocating funds among activities and modifying the results indicators. The restructurings, which created a dedicated project implementation unit, permitted to allocate more funds to the more successful activities, and produce more outputs than envisaged at the start of the project with a smaller expense. The restructuring, though, did not lead to better collection and analysis of information about project costs per unit of output and project outcomes. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The design was ruled by the M&E arrangements established in the Action Plan to Strengthen and Modernize Public Finance Management, Phase II (known as PAGAM II). The first plan, established in 2005,and renewed in 2010 (hence PAGAN II), was an essential element of Mali’s Poverty Reduction Strategy Papers. The project’s outcome and results indicators were derived from PAGAM II, which had established a monitoring framework. Neither the PAD nor the ICR specified what M&E arrangements were taken from PAGAM II and applied to the project. The PAD said that the Unit for the Support of Public Finance Reform (CARFIP) had capacity in monitoring and evaluating programs, and that assessing progress in implementing the project and its results would be undertaken regularly in the context of the PAGAM II. The intermediate results indicators were well aligned with the activities the project financed, but the links to the second and fourth project objectives were not specified; moreover, from the intermediate results and their related outcomes it was not possible to conclude if budgetary decision making and oversight and transparency had improved. In summary, the results chain was inappropriate to assess completely the project outcomes. b. M&E Implementation The security situation affected the implementation of the monitoring arrangements at the beginning of the Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) period, a problem that the lack of a properly established implementation unit exacerbated. The ICR notes that after the security situation stabilized and an M&E specialist was recruited, supervision started in a regular manner and timely reports were produced. Although these activities were carried out, they are not reflected in the information supplied by the ICR about results and project costs per activity and the timeline of the results achieved. Also, the improved situation and the two project restructurings did not lead to results indicators that could help measure in a proper way whether objectives 2 and 4 were achieved, as discussed in the efficacy section. c. M&E Utilization After the initial setbacks, progress in project implementation and data gathering, with some limitations in their amount and quality, influenced the two restructurings, which covered both the activities carried out and the indicators used to measure intermediate and objective results. There was no information gathered that could help assess the impact of the project on budgetary decision making and oversight. M&E Quality Rating Modest 10. Other Issues a. Safeguards None triggered. The project was rated Environmental Assessment Category C. b. Fiduciary Compliance All conditions were met, including the adoption of a Project Implementation Manual and the contracting of a financial management specialist. The ICR presents little information about procurement and financial management for the project accounts. c. Unintended impacts (Positive or Negative) None, in this review's view. The ICR discusses unintended impacts in section IIE. It repeats in a longer and different manner the results reviewed under efficacy in section 4. d. Other --- Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The ICR rates objective 2 Moderately higher than it should for the Outcome Satisfactory Satisfactory evidence it presents calls forth. Efficiency was modest. The ICR’s combined ratings for quality at entry and quality Moderately Bank Performance Satisfactory of supervision should lead to Satisfactory an MS rating, not to an S rating Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR offers the following lessons: • Project management units overloaded with projects or responsibilities for reform are unlikely to fulfill all their responsibilities for every project or reform under their purview. To prevent problems in project implementation it is important to set up a project implementation unit dedicated to carry out all the activities of the project. • In countries affected by conflict and violence, decentralization projects can achieve outcomes through a focus on building capacity and improving the dialogue between the Government and citizens through the adoption of adequate practical tools, such as citizens' budgets or citizen surveillance committee reports. • Successful and timely implementation of project activities depends on adequate levels of technical assistance. The long absence of a knowledgeable procurement specialist contributed to delaying the implementation of this project; when the Bank provided a qualified procurement professional to assist the country, it was possible to remove technical bottlenecks and expedite the execution of the project. • Building capacity must be an on-going process in countries with low capacity and frequent rotation of qualified government staff. Doing so is necessary to ensure that the technical knowledge of the qualified staff is preserved even when the personnel rotates frequently. The IEG review offers the following lessons: • One size is unlikely to fit all. The experience of this project shows that a framework prepared for monitoring general reforms in public financial management should be complemented with a project-specific framework that captures the characteristics and nuances of the design and implementation challenges of each individual project. • Lofty objectives require proper results indicators. This project's experience indicates that a good results framework is necessary to assess project outcomes properly; that requires ensuring that the interventions Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ML-Governance and Budget Decent. TA Proj (P112821) and results indicators lead to achieving the project’s objectives, even broad and lofty ones (e.g., 2, 3, and 4). 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR presents all the relevant aspects of the project in a clear and complete manner. It is noteworthy in its presentation of the theory of change and the rationale for the project (pp. 7, 8, and 46) and its effort to analyze efficiency, despite its shortcomings, but also the limited treatment it gives to administrative aspects of the project such as procurement and financial management. Surprisingly, the ICR does not delve in more detail into project costs and costs per unit of output. On this matter, the ICR does not inform on the actual cost of the project, both in its external and internal components; as it stands, it seems that all costs were financed by the credit, and the government (central and local) contributed nothing to it. The ICR could have updated the information on the Budget Index rating to 2017. The information in Section IIE basically repeats what the ICR discusses in the efficacy section. The ICR uses acronyms and abbreviations heavily. That use requires their scrupulous compilation in the ICR’s acronym page; on this score, the ICR fell short, slowing down the reading of the document. a. Quality of ICR Rating Substantial Page 15 of 15