Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) Report Number: ICRR0021793 1. Project Data Project ID Project Name P147343 ZM-Public Fin Mgt Reform Program Phase 1 Country Practice Area(Lead) Zambia Governance L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-17118 31-Dec-2017 22,000,000.00 Bank Approval Date Closing Date (Actual) 06-May-2014 31-Dec-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 22,000,000.00 22,000,000.00 Revised Commitment 22,000,000.00 22,000,000.00 Actual 22,000,000.00 22,000,000.00 Prepared by Reviewed by ICR Review Coordinator Group Zeljka Sedlo Stucka Judyth L. Twigg Malathi S. Jayawickrama IEGEC (Unit 1) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The overall Program Development Objective (PDO) was to “contribute to improving the efficiency and accountability of public resources." The PDOs were consistently used throughout the background documentation (Project Appraisal Document, Grant Agreement, and ICR). A December 2017 restructuring added and dropped some PDO-level indicators, but these reflected changes in measurement rather than in the project's scope or ambition. A split rating is therefore not warranted. Page 1 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project entailed eight components: C1: Integrated public planning and budgeting (appraised US$ 1.87 mil. vs. actual US$ 1.37 mil.) to improve planning and budgeting processes through three subcomponents: 1) review of the 2004 Public Finance Act and laws and regulations on fiscal decentralization, management of public debt, management of government investments, audits, and monitoring and evaluation; 2) support for initiatives of the Budget Department to establish better linkages to planning; and 3) pilot-testing and the potential rolling out of a budget module in the Integrated Financial Management Information System (IFMIS). C2: IFMIS and cash management (appraised US$ 7.64 mil. vs. actual US$ 5.26 mil.) to enhance financial management systems and standards by promoting good governance standards to ensure: 1) timely budget execution; 2) publishing of good quality financial reports; and 3) enhanced cash management efficiency and lower borrowing costs. C3: Public procurement reforms (appraised US$ 4.06 mil. vs. actual US$ 3.68 mil.) to strengthen oversight and capacity in public procurement by: 1) establishing oversight of the Zambia Public Procurement Authority (ZPPA); and 2) strengthening the capacity of the Zambia Institute of Purchasing and Supplies (ZIPS). C4: Enhancing internal audit and control (appraised US$ 1.70 mil. vs. actual US$ 1.25 mil.) to promote internal control systems by: 1) developing a curriculum for internal auditors; 2) upgrading assurance procedures to international standards; 3) strengthening audit committees; and 4) promoting the use of technologically advanced audit tools and skills. C5: Increasing the effectiveness of revenue administration (appraised US$ 10.08 mil. vs. US$ 7.64 mil.) by the Zambia Revenue Authority (ZRA). This was the most complex component, consisting of seven sub-components aiming at implementing: 1) a mining sector monitoring mechanism; 2) a system for monitoring the balance of payments; 3) a transit tracking and suspense regime; 4) a capacity building program for the Taxpayer Office; 5) investigation of internal affairs, 6) bulk intelligence data analysis and dissemination of tax/custom discrepancies; and 7) electronic records management system and scanning methodology in the Central Registry. C6: Strengthening tax policy formulation (appraised US$ 0.58 mil. vs. US$ 0.43 mil.) to enhance the tax policy capacity unit and support: 1) studies and research on tax policy formulation; 2) training and mentoring; and 3) access to information to aid tax policy formulation. Page 2 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) C7: Strengthening program management (appraised US$ 1.52 mil. vs. US$ 1.26 mil.) to support procurement, financial management, and monitoring in implementing departments to help monitor objectives and performance. C8: Employing other public financial management (PFM) interventions (appraised US$ 2.55 mil. vs. actual US$ 1.11) to support: 1) the Office of the Auditor General to ensure better efficiency and effectiveness; and 2) the Zambia Institute of Chartered Accountants to enable the implementation of a revised Chartered Accountant qualification with a focus on public sector accounting streams. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project financing and cost: The project was financed from the Multi-Donor Trust Fund in the total amount of US$ 22 million. An informal change in project components and project costs (ICR, table 4, p. 14) occurred on February 18, 2018. The original project was appraised at US$30 million and was reduced to US$22 million due to exchange rate fluctuations and a reduced commitment (GBP2.35 million) by the Department for International Development (DFID). DFID could not extend the project due to its internal budget cycle. Borrower Contribution: The government contributed to financing through counterpart funding in the amount of US$2.83 million. Dates: The project was approved by the Board on May 6, 2014 and became effective on July 17, 2014. The grant closing date was extended (from December 31, 2017 to December 31, 2018) to allow for completion of critical project activities: the implementation of an electronic procurement system, rolling out of output- based budgeting, and development of a public investment management framework. The project was restructured twice: 1) on December 29, 2017 to extend the closing date and modify the results framework; and 2) on February 18, 2018 to amend the implementation schedule and change component costs. 3. Relevance of Objectives Rationale The Zambian economy was rapidly expanding at the time of project appraisal. The government’s main challenge at that time was to ensure prudent fiscal policy and macroeconomic stability. The project’s objectives were aligned with the government's strategic vision outlined in the Sixth National Development Plan (2011-2015) and in the revised Sixth National Development Plan (2013-2016). The vision was to strengthen overall efficiency of public sector management by implementing reforms for more efficient revenue mobilization and public spending. During the project’s lifecycle, the government developed a three- year (2013–2015) PFM Reform Strategy focusing on improving efficiency, effectiveness, accountability, and transparency in the use of public resources. The PDOs were also aligned with the Bank's Country Partnership Strategy at appraisal (2013-2016), which emphasized the need for improving governance and economic management and thus strengthening systems and processes for public sector performance. Page 3 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) The Bank's recently approved Country Partnership Framework (CPF, 2019-2023) continues this legacy with building resilience in Zambian public institutions as one of its three core strategic areas. The CPF objective 3.1 (fiscal and financial fitness) reflects the Bank's commitment to supporting Zambia in further reform of PFM. Similarly, the government’s current Seventh National Development Plan and Vision 2030 also emphasize the need for continuous strengthening of public sector institutions. The continued implementation of PFM reforms should foster economic development, poverty reduction, and improved public service delivery. Finally, the Bank’s Africa Region Strategy emphasizes the need to build governance and public sector capacity through public expenditure management systems. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Contribute to improving the efficiency of public resources Rationale The objective was to be achieved by 1) improving budget credibility and fiscal discipline, 2) enhancing revenue mobilization and administration, and 3) strengthening internal controls (ICR, Theory of Change, para 7). Budget credibility improvements were planned through the introduction of three IFMIS modules (planning, budgeting and cash management) and Treasury Single Account (TSA). Enhancements in revenue mobilization were to be facilitated through the introduction of i) a mining value chain monitoring mechanism, ii) a tax invoice management system (TIMS) and iii) cargo transit tracking. Finally, strengthening of internal controls was to be achieved by the implementation of an Audit Management System (AMS). By the end of the program, the following outputs were achieved: IFMIS modules were introduced in 51 ministries, provinces, and spending agencies (MPSAs), which exceeded the target of 36 MPSAs. The TSA was introduced and rolled out to 51 MPSAs, which improved cash management and processing of government payments. The TSA coverage increased to 80 percent of all budget disbursements (ICR, Annex 1). In addition, 74 percent of government payments are now made through the TSA. The ICR claims that consolidated cash balances ease budget execution and help avoid (more expensive) short-term borrowing. There was no attempt to measure potential savings resulting from the real-time information on the availability of funds. Page 4 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) The Mining Value Chain Monitoring Mechanism was made operational in twelve large mines. As of end- December 2018, 65 small-scale mines also started submitting Mineral Production Reports using the Mineral Output Statistical and Evaluation System. The introduction of these systems boosted fiscal receipts from the mining industry: mineral royalties grew by 61 percent between 2017 and 2018 (K3,936 million, up from K2,435 million) and company income taxes increased by 120 percent (K2.356 million up from K1,067 million). Taken together, these equate to a revenue increase of 1 percent of GDP (ICR, para 22 (i)). The TIMS was introduced. As a result, amounts of net value-added tax (VAT) declared and recorded appear to have increased. The ICR (para 22 (iv)) mentions an increase in the net VAT from March 2018 to September 2018 compared to the same period in 2017. However, the ICR does not support this statement with specific data. Tax training and new policies and procedures optimized revenue collections. In 2018, the total recovered debt (tax arrears) increased five-fold compared to 2016 (ICR, para 22). A Cargo Transit Tracking system was launched, and it is expected to help in preventing illegal diversions and revenue evasion. The ZRA is now able to track high-risk goods (ICR, para 22 (ii)). The ICR does not indicate the potential savings from the introduction. A PIMS was created. The ICR (p. 16, para 2) states that the system enables a comprehensive analysis of government investment projects and optimizes investment costs. The project did not attempt to measure the scope of cost savings. An AMS was introduced to enhance internal audit processes and shorten time needed for audits. Several audit related outputs were mentioned in the ICR, such as training on performance, information technology audits, risk audits, assessments of audit quality, and preparations of audit manuals (ICR, pg. 17, para 7), but the ICR does not cite the number of trainings or specify participants who received the training. Outcomes Key outcome targets, as measured by changes in Public Expenditure and Financial Accountability (PEFA) scores from 2012 to 2018, were met: Aggregate expenditure outturn versus originally approved budget (PI-2) improved from the baseline D, achieving the target of C. Effectiveness of internal controls (PI-21) improved from the baseline C+, achieving the target of B. Revenue targets announced for each year were achieved (ICR, Annex 1). These indicators are reasonable measurements of improved efficiency in the use of public resources. Achievement of this objective is therefore rated Substantial. Rating Substantial OBJECTIVE 2 Objective Contribute to improving the accountability of public resources Rationale Page 5 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) The objective was to be achieved through increased transparency in public finance and transparency in procurement. By the end of the program, the following outputs had been achieved: The PFM Act 2018 and Public Procurement Law were passed by the Parliament, expected to strengthen accountability and transparency (ICR, pg. 5, Executive Summary). An e-government procurement (e-GP) system was developed and introduced in a total of 37 procuring entities, exceeding the target of six entities. The e-GP is considered to represent a new era of transparency in procurement processes. The mineral value chain monitoring is reported to have brought transparency and accountability to the mining sector (ICR, para 22(i)). However, there is no specific evidence provided. Zambia Chartered Accountants qualification program is reported to have strengthened the accountancy profession. The ICR states (pg. 5, Executive Summary) that the revised chartered accountancy qualification provides a basis for transparency and accountability of budget management, but no specific evidence is provided to support this claim. ZIPS regulations were prepared and submitted for legal review, and a professional education curriculum and training courses were implemented. The ICR (para 26) states that Zambian students now have the option of obtaining sound professional qualifications in public procurement at a fraction of the cost when compared to global certifications. The project also resulted in several positive, though unplanned results. First, the e-GP was intended to be piloted for only six ministries, but because of its success it has been rolled out to more than 20 ministries. Second, the payroll system was migrated from limited capacity hardware to a new server hosted in the G- Cloud which has stabilized payroll operations and improved budget controls. Outcomes Key outcome targets, as measured by changes in PEFA scores from 2012 to 2018, were not met: Transparency in public finances (PI-24, budget reporting) improved from the baseline of C to B, not achieving the target of B+. The ICR explains that the shortfall was due to delays in uploading reports to the Ministry of Finance website. Transparency in public procurement (PI-19, competition, value for money, and controls in procurement) improved from the baseline of D+ to C+, not achieving the target of B. In addition to not meeting targets, these indicators measured transparency but did not fully measure the objective of accountability. Achievement of this objective is therefore rated Modest due to insufficient evidence of impact. Rating Modest OVERALL EFF TBL Page 6 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) OBJ_TBL OVERALL EFFICACY Rationale The project substantially achieved the objective to contribute to improved efficiency in use of public resources, but only modestly achieved the objective of improved accountability. Overall efficacy is therefore rated Substantial, but with moderate shortcomings. Overall Efficacy Rating Substantial 5. Efficiency Economic analysis: The PAD highlighted the project’s benefits. Efficiency analysis for PFM projects is challenging, as some benefits -- the results of technical capacity building, new legal frameworks, or the introduction of a new system that has yet to become fully operational -- are difficult to quantify. However, the task team could have applied stronger quantitative analysis in addition to descriptive results. The ICR cites benefits from some reforms without providing supporting evidence, such as cost savings without corresponding dollar amounts and reduced time in processing transactions without hours/days saved. Implementation efficiency: The ICR cites evidence for implementation efficiency. The project was executed within the approved cost envelope, and the one-year extension had no monetary implications. There was strong and seemingly efficient financial management and procurement (ICR, paras 62, 63), and appropriate sequencing of project activities (ICR, paras 46, 47). However, efficiency was somewhat impacted by challenges involving government commitment and capacity challenges during the project’s first eighteen months (paras 50, 52), staff changes in government (para 54), and issues with intra-ministerial coordination and with consultants (para 71). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable Page 7 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project's objectives remain highly relevant to country context, government strategy, and Bank strategy. The project substantially achieved the objective to contribute to improved efficiency in use of public resources, but only modestly achieved the objective of improved accountability. Although a formal economic analysis is not provided and there were moderate implementation inefficiencies, project efficiency is rated Substantial. Given moderate shortcomings in the project's preparation and implementation, Outcome is rated Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Sustainability of the achievements of PFM reforms will vary among institutions and will greatly depend on budgetary constraints, technical capacity of staff, presence of donor support, and political will to continue implementing reforms. The ICR (para 70) states that poor budget credibility could weaken implemented PFM reforms. With insufficient cash and poor budget execution, newly implemented IFMIS cash management and budget execution modules may become underutilized. The risks vary by government agency. PIMS is a new unit without funding and with limited technical capacity. ZPPA employs staff with limited experience on e- procurement. The IFMIS requires continued technical support in using the implemented modules and expanding coverage. TSA may lose momentum without donor support, as it needs to expand to ministries at district level that control cash outside the TSA. There is no institutional structure in place for International Public Sector Accounting Standards. ZIPS relies on self-financing and may need external resources. 8. Assessment of Bank Performance a. Quality-at-Entry The project was complex with an ambitious agenda: to set solid foundations for the public finance sector. In this regard, the project was designed to meet the authorities’ multifaceted needs. The project used analytical work carried out under PEFA 2012 (PAD, para 7) and was designed around a phased approach to PFM reform (ICR, para 64). In line with this approach, the project team developed a sequenced strategy and outlined a detailed action plan for each government agency. However, just few months after implementation started, the project team had to overcome obstacles such as shortage of Page 8 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) key staff, a disconnect between government and Bank procurement procedures and processes, and a lack of clear strategy and plans from government side. Those shortcomings had to be addressed in the first 18 months, and they triggered a revision of PDO indicators to better align targeted outcomes with project components. Resident Advisors funded by the German International Development Agency were available to provide support on internal audit and budgeting components. An IFMIS Specialist was hired with donor support to provide real-time advice. The Bank implementation support team included audit, procurement and financial management specialists. A resident US Treasury Advisor also provided guidance on the TSA. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project was closely monitored by the task team lead (TTL), and initially identified risks were promptly addressed. For instance, the ICR (para 57) describes two M&E-related shortcomings that surfaced in the initial phase -- a disconnect between the indicators and data collection, and lack of an M&E expert -- that were promptly addressed. Consequently, the team worked with donors’ technical teams that supported the Project Management Unit (PMU) with technical expertise to implement a solid M&E framework. The project was monitored in close collaboration with other donors (ICR, para 47). In addition, global technical experts provided inputs to project components during mini-missions. All sub-projects were designed and executed under the guidance of the same TTL, who remained responsible throughout the project lifecycle. This appears to have been a key element of building strong partnerships with the government and donors through a Joint Government Donor Committee. Regular reports were prepared by the PMU and the task team. Finally, the ICR (para 57) notes that the project was reviewed externally on a regular basis, though it does not identify the reviewer or summarize any findings. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The Public Financial Management Reform Unit (PFMRU) of the Ministry of Finance was responsible for data collection and analysis. The M&E design at first proved to be a challenge for this project. The original outcome targets outlined in the PAD were poorly aligned with the eight project components, which triggered revisions during the project’s mid-term review (May 2016). Six new intermediate outcome indicators were Page 9 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) added, and one outcome indicator was dropped. Overall, the attempt to use PEFA indicators as a measure of progress, in the absence of a better quantitative-based assessment, was challenging in terms of relevance and measurement (ICR, para 71 (xi)). . b. M&E Implementation The PFMRU collected and reported data on project indicators as planned. Results were used to inform project decision-making, including when the project had to be restructured (extension of closing date and revised indicators). The M&E capacity built during the project is envisioned to be utilized in future projects in Zambia. c. M&E Utilization The M&E data and reports were fully utilized to inform decision-makers regarding project activities (ICR, para 58). First, the data and information collected from the mid-term review was used to restructure the project and extend its closing date. Second, modifications of annual work/procurement plans were conducted whenever required and based on progress reports. Third, the Bank used the evaluation data to reconfigure the type of technical assistance needed for the project (hiring of an output-based budgeting expert, professional accounting body experts, and tax policy experts). M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was rated Environmental Assessment category C and did not trigger any environmental or social safeguards. b. Fiduciary Compliance The project complied with fiduciary agreements during implementation. The project enabled policy reforms and strengthened controls in Zambia’s PFM. Hence, the project reduced key fiduciary risks in the PFM system. The ICR (paras 62 and 63) states that both financial management performance and procurement performance were satisfactory. Financial management (FM) performance: The FM Manual incorporated necessary internal controls using country finance systems, which worked as planned. Key staff attended a training workshop to become familiar with the Manual. Quarterly interim unaudited financial reports were submitted to the Bank on time. Page 10 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) Audits for the project were conducted by the Office of the Auditor General and provided an unqualified audit opinion on the financial statements. Procurement performance: Procurement packages were completed according to the procurement plan. All prerequisites were in place to ensure successful procurement and implementation. Implementing agencies under the Ministry of Finance (through its various departments), the ZPPA, the ZRA, and the ZIPS maintained professional staff experienced in similar reforms and familiar with donor policies and procedures. Agencies implemented procurement contracts using competitive methods. Procurement contracts were awarded in line with the Bank’s procurement guidelines. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Modest achievement of improved accountability in the Outcome Satisfactory Moderately Satisfactory use of public resources, and modest efficiency. Shortcomings at entry led to a Bank Performance Satisfactory Satisfactory need for restructuring after 18 months of implementation. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The main lesson from this ICR relates to the usage of PEFA scores as PDO indicators. PEFA scores are not straightforward and they are challenging to measure. Meticulously described PEFA scores with an emphasis on actions that need to be undertaken for an upgrade/downgrade would help strategize better in achieving targets. Page 11 of 12 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ZM-Public Fin Mgt Reform Program Phase 1 (P147343) 13. Assessment Recommended? Yes ASSESSMENT_TABLE Please Explain The project aimed at improving the efficiency and accountability of public resources through a multifaceted project design that consisted of eight components. Each component corresponds to at least one pillar of public finance reform, which together formed a programmatic approach to reform. An assessment would evaluate benefits and weigh cost-risks of such an approach compared to other approaches. This project won a VPU award in FY19, and it may be worth exploring whether the experience and lessons learned by the task team may be used as an example of PFM project design in peer countries with similar backgrounds. 14. Comments on Quality of ICR The ICR is well written. It provides a plethora of project details, clearly describes project components, and summarizes key outputs. The write-up does not shy away from describing implementation shortcomings and from discussing clearly the obstacles that caused the project extension and affected its performance. There were moderate shortcomings: the efficacy discussion is organized by agency and activities, rather than by achievement of objectives. Although it highlights the shortcomings with PEFA scores as PDO indicators, it does not explore alternatives, and it does not present alternate sources of evidence for project achievement. a. Quality of ICR Rating Substantial Page 12 of 12