ICRR 14208 Report Number : ICRR14208 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/27/2013 Country : Zambia Project ID : P082452 Appraisal Actual Project Name : Public Sector US$M ): Project Costs (US$M): 100 108 Management Program Support Project L/C Number : C4140 Loan/ US$M ): Loan /Credit (US$M): 30 27 Sector Board : Public Sector US$M): Cofinancing (US$M ): 70 81 Governance Cofinanciers : African Development Board Approval Date : 01/05/2006 Bank; DANIDA; SIDA; Closing Date : 12/31/2010 06/30/2012 Irish Aid; European Commission; NORAD; DfID; FINIDA Sector (s): Central government administration (96%); General finance sector (2%); Sub-national government administration (2%) Theme (s): Public expenditure; financial management and procurement (29% - P); Administrative and civil service reform (29% - P); Law reform (14% - S); Other public sector governance (14% - S); Debt management and fiscal sustainability (14% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Robert Keyfitz Rene I. Vandendries Ismail Arslan IEGPS2 2. Project Objectives and Components: a. Objectives: The project development objective (PDO) in the Project Appraisal Document (PAD) is identical to the Loan Agreement apart from for minor differences in wording . From the PAD: “The project development objective is to make public service delivery institutions and processes more effective and efficient by enhancing the efficiency and accountability of the management of public expenditures and the public service in line with the GRZ [Government of the Republic of Zambia ] strategies to support poverty reduction and promote economic growth .� The PDO in the results framework (Annex 3) adds adoption of direct budget support to the above : “To improve the quality of public expenditure management and public service effectiveness in Zambia as a means to achieve: 1. Economic growth and poverty reduction; 2. A more efficient public service; and 3. The adoption of direct budgetary support as the principal mechanism for development aid in Zambia . “ This review follows the ICR in using the PDO from the PAD, since mechanisms for development aid pertain to development partners rather than the Government and besides are not represented in the results framework . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project was the planned second phase of a 13 year Adaptable Program Loan (APL) launched in 2000, intended to support the Government's Public Sector Management Program (PSMP). It comprised three major components: (1) Public Expenditure Management and Financial Accountability (PEMFA): Support for omnibus PEMFA reform addressing specific institutional and capacity deficits . Twelve subcomponents were broken out, with key elements: Rolling out automated financial management systems to all central spending agencies and provinces, incorporating new treasury management and accounting systems . Reforming budget preparation and execution, through implementation of policy based budgeting, improved economic and fiscal analysis, more transparency in budget documents, improved cash management and expenditure monitoring. Strengthening debt management, including legal reform, capacity building, compiling data, adoption of a debt management strategy, and development of a public debt market . Improving internal and external audit, through capacity building, developing internal control systems, and strengthening the audit profession more generally . Enhancing parliamentary oversight, by developing skills and awareness of the relevant parliamentary committees and strengthening linkages with the auditor general, and promoting public participation . Reforming public procurement through a revised Procurement Act, guidelines and code of conduct, development of specialized skills and oversight . (2) Public Service Management (PSM): Right-sizing the public service to sustainable levels and reforming pay to enhance performance and quality of service delivery . Improving payroll management and establishment control . (3) Support for Retrenchment: Funding severance pay and social safety net provision for retrenched staff . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The operation was approved by the Board on January 5, 2006 and became effective on May 10, 2006. In December 2010 a restructuring extended the initial closing date from December 31, 2010 to June 30, 2012. A large group of development partners cofinanced various parts of the first two components, while the Bank alone financed the third component . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Zambia has recognized the importance of PFM and public sector reform since the early 1990s, and in 2004 the Government had initiated its Public Sector Management Program (PSMP) which the present operation supported (hence, the PSMP Support Project). Public sector reform is especially important for Zambia in light of managing resource (copper) revenues, and is closely aligned with Zambia ’s long term development strategy, Vision 2030, as well as the fifth National Development Plan . The project responded to a Government request for the Bank and other development partners to harmonize and coordinate their support in a SWaP . PEMFA and PSM remained high priorities in the Bank’s CAS at closing (FY08-FY11). Indeed, CAS Outcome 2.1, "Strengthened public financial management, procurement and oversight capacity" probably referred to the project and incorporated many of its elements, including the Integrated Financial Management Information System (IFMIS) roll-out, public service management and procurement reform, and strengthening of audit capacity. The CAS also emphasized harmonization and coordination among Zambia’s main development partners, as exemplified by the project . Relevance of objectives :high b. Relevance of Design: The Bank collaborated closely with GRZ and other development partners during preparation and substantive reforms were based on the first phase of the APL as well as other diagnostic work including a PEMFA report undertaken by the Government and various studies by other partners . Nineteen subcomponents were allocated to various partners to be supported with technical assistance, training, and IT investment . Much attention was given to the project's administrative structure in order to pool funding and harmonize support . PEMFA and PSM Secretariats were established in the Government to manage the first two components (though they were later disbanded), with oversight and coordination by a steering committee made up of senior government officials . Joint technical working groups (JTWGs) were established to liaise between the Government and the cooperating partners. The design was detailed and thoughtful, but overambitious and far too optimistic about the government ’s willingness and capacity to implement it . Instead of coordination, the elaborate administrative structure led to endless and indecisive meetings, procurement delays, and the commissioning of consultants' reports which were never implemented. At a substantive level the project remained quite decentralized with various subcomponents proceeding largely independently and motivated by donors' preferences as much as the Government’s institutional capacity. Sequencing was not well planned and inadequate attention was given to capacity building needed for the project itself . Moreover, the reforms had already been underway in a piecemeal fashion under the terms of a 2004 Memorandum of Understanding (MoU) and there was little clarity about the Bank's role and value added, or how its participation would enhance overall coordination and harmonization . In practice, the Bank was not able to exert much authority . At the midterm review, which was conducted by the Government, the scope of the project was narrowed and some subcomponents were consolidated or dropped in order to focus on priorities. The Bank did not control or approve the changes and no restructuring paper was produced. Thereafter, implementation continued to lag and outcomes were mixed . Results framework: PDO indicators in the main text of the PAD differed from the results framework in Annex 3. However, both focused on outputs rather than outcomes, and were only tenuously linked to the PDOs of public sector efficiency, effectiveness and accountability . The PAD version was somewhat better, e .g. including the deviation between actual and budgeted spending, but the ICR reported only the Annex version as did the ISRs . As a result, the project's PDOs were not well measured . The results framework also comprised 58 intermediate outcome indicators -- at least 3 for most of the 19 subcomponents, and as many as 9. However, the ISRs reported on only 14 of these, so few that a few subcomponents were not covered at all, including some which represented major expenditures . Rating for design relevance : modest 4. Achievement of Objectives (Efficacy): Measured in terms of outputs the project would seem to have achieved significant progress, with most of the indicators tracked in the results framework fully or partially achieved . Allocating indicators across subcomponents, the ICR concludes that at least partial success was realized on most of these as well . Efficacy in terms of objectives is less clear, in part because the PDO indicators are not clearly linked to objectives and in part because the pooled funding and shared responsibility make attribution difficult . However, three factors suggest a cautious assessment : (1) Because implementation lagged, many of the project's achievements occurred only near closing, hence would not have had time to take effect; (2) The reforms were complex and interlinked so that in many cases failure in one area would have rippled more widely; and (3) Many results indicators were formally satisfied without translating into practical, behavioral change . The project’s objectives do not align with its components and this review follows the ICR in decomposing the PDO into three elements: Make public service delivery institutions and processes more effective and efficient; Enhance the efficiency of public expenditure management; and Enhance accountability in the public service . Make public service delivery institutions and processes more effective and efficient : MTEF and Activity Based Budgeting (ABB) were formally implemented throughout the government and major parastatals, and linked to strategic planning and operations, even exceeding the target .However, some results were achieved only within a month of the project closing, and some legal reforms, staffing and training are still pending . More generally, wide experience with MTEF strongly suggests that more time will be required for such a major institutional change to take hold. Rating : Modest Enhance the efficiency of public expenditure management : A new Finance Act was passed, strengthening the legal basis for public expenditure management, though not all PEMFA supported reforms were included and the Act was given approval only a month before the project closed . Debt management was strengthened, donor support integrated into the budget process, and internal audit functions were computerized, however further training will be needed to put new tools and procedures to work . Payroll and establishment controls were put in place and computerized, and some staff were retrenched though much less than expected . Performance management has not yet been linked to strategic planning and procurement reforms have only been partially implemented. Rating : Modest . Enhance accountability in the public service : New regional offices were built to allow the Office of the Auditor General to execute its accountability role, but they have not yet been fully staffed with qualified personnel . Central ministries prepare 5 year strategic plans, but these are not yet linked to performance -based pay as required. Activity based budgeting has been partially implemented . Committee rooms were built to facilitate parliamentary oversight, but it is not clear whether the parliamentary oversight has improved . Rating : Modest 5. Efficiency: Because of pooled funding and joint responsibility and the continuing role of the 2004 MoU in guiding project activities, it is not possible to calculate efficiency . Nevertheless, given the moderately unsatisfactory outcome it is doubtful rates of return on the project represented an efficient use of funds . Efficiency rating: Modest ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Although the project's objectives were highly relevant, the design was only modestly relevant to actual conditions. Overly complex project management arrangements made coordination difficult, and the results framework did not facilitate monitoring. While many outputs may have laid important groundwork for future reform efforts to build on, efficacy in achieving the project's objectives was mostly modest and efficiency, though impossible to measure accurately, is rated as only modest as well . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: It remains to be seen whether the project's objectives accord with the priorities of the new Government elected in late 2011. However, it does not bode well that cabinet approval has been pending for some elements of the project for two years or more. Meanwhile, administrative changes have proved disruptive, with realignment of Ministries and frequent changes of personnel having project responsibilities -- 6 Permanent Secretaries in the Ministry of Finance, 4 Accountants-General, and 4 IFMIS Project Managers. Uncertainty about political will, a general lack of capacity and administrative instability suggest the risk is considerable although at some point the project's achievements may be seen as having represented a useful step forward . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project was well-intentioned but over ambitious and the design was flawed . The proposed reforms were too broad and all-encompassing and did not pay enough attention to sequencing, or roles and responsibilities of the Government and other cooperating partners . During preparation and appraisal the team felt it had limited scope to revise the main parameters of an operation that was already underway, but underestimated the negative effect of some key issues on the achievement of results . A poorly designed results framework further impeded effective monitoring of progress at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Normal supervision missions were not fielded regularly . Instead the Bank relied on review mechanisms agreed in the 2004 MoU which relied on various committees and working groups under the Government's overall leadership. More assertive supervision might have enhanced the Bank's credibility with both the Government and other partners. Lengthy intervals between ISRs and overly optimistic ratings compromised their use as a reporting tool, and communication with the project TTL based in country appears to have generally been poor. As a result, Bank management may not have been adequately informed about the project's status. No restructuring paper was prepared after the Government made major changes to the composition of the project at the 2008 mid-term review. This might have prompted an in-depth review of the project leading to closer supervision or even early cancellation . Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: An uneven commitment as illustrated by lengthy delays at the cabinet level in taking required actions to support the project was detrimental to achieving the objectives . Limited capacity and high levels of staff turnover further impeded results Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: A large number of government bodies were involved in some way with project implementation, some established specifically for the project, others with more general responsibilities such as the Ministries of Finance and Justice, the Office of the Auditor General, the Zambia National Tender Board, the Parliament, etc. Some performed better than others judging by the varying degrees of success in achieving outputs . Nevertheless, there were continual concerns with record keeping, oversight and accountability, as well as high levels of staff turnover . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: As discussed above, PDO indicators did not bear on the PDOs, making overall progress hard to track, though a list of 58 intermediate indicators proposed in the results framework would have adequately covered the project's outputs. The PEMFA and PSM Secretariats in charge of the first two components included M&E specialists to be tasked with preparing quarterly reports for distribution to the cooperating partners . In addition regular supervision missions by the Bank were planned . b. M&E Implementation: M&E arrangements proposed in the 2004 MoU appear to have substituted for the Bank's standard supervision judging by the lack of Aides Memoires and other documentation filed in the system . ISRs were not prepared in a timely fashion, reported on only 14 of the 58 proposed intermediate indicators, and may have been overly optimistic in assigning ratings. Thus, Bank management may not have been kept adequately informed about the project's status. c. M&E Utilization: Weakness in the M&E framework and its implementation likely contributed to the Bank's marginalization in the project and contributed to the overall moderately unsatisfactory outcome . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: No issues raised b. Fiduciary Compliance: The ICR notes a number of fiduciary issues, including : (1) late submission of unaudited interim financial reports and inadequate explanations of variances between budgeted and actual expenditures; (2) weak internal audit controls; (3) payments made before the Bank issued non -objections; (4) slow disbursement, for an eventual total of only 85% of the credit; and (5) undocumented and ineligible expenditures which the Government subsequently reimbursed. c. Unintended Impacts (positive or negative): d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR draws a number of lessons about the importance of Government ownership and capacity, realistic objectives, and adequate risk mitigation . It urges caution in entering into government -led, multi-donor, pooled funding arrangements, especially pertinent in this case where the project was already in progress according to a MoU signed 18 months before the project was approved and remained in force, effectively tying the Bank's hands. Another important lesson concerns the need for better support and mentoring for new TTLs, especially as in this case a locally hired one who was not familiar with the Bank's complicated and sometimes obscure internal procedures. Closer supervision of the TTL might well have secured a better outcome . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: Preparation of the ICR was hampered by a lack of institutional continuity -- staff turnover in the Government and development partners and the project TTL's departure from the Bank -- combined with weak and poorly documented supervision. Despite these problems, the ICR gives a credible and insightful representation . a.Quality of ICR Rating : Satisfactory