Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004953 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF A4518 ON A SMALL GRANT IN THE AMOUNT OF USD 0.497 MILLION TO THE Republic of Nicaragua FOR Improving Quality and Efficiency of Public Sector Audit (P161359) June 5, 2020 Governance Global Practice Latin America And Caribbean Region Regional Vice President: Humberto Lopez, Acting Country Director: Yaye Seynabou Sakho Regional Director: Robert R. Taliercio Practice Manager: Xiomara A. Morel Task Team Leader(s): Lourdes Consuelo Linares Loza ICR Main Contributor: Lourdes Consuelo Linares Loza . ABBREVIATIONS AND ACRONYMS ASF Mexico Supreme Audit Institution (Auditoria Superior de la Federación) CGR Office of the Comptroller General CPF Country Partnership Framework GoN Government of Nicaragua GRS Grievance Redress Service ICR Implementation Completion Report IDB Inter-American Development Bank IDI INTOSAI’s Development Institute IR Intermediate Result ISR Implementation Status and Results Report ISSAIs International Standards for Supreme Audit Institutions INTOSAI International Organization of Supreme Audit Institutions IT Information and Technology MHCP Nicaragua’s Ministry of Finance and Public Credit (Ministerio de Hacienda y Crédito Público) MIFIC Nicaragua’s Ministry of Development, Industry and Trade (Ministerios de Fomento, Industria y Comercio) NAGUN Nicaraguan Government Auditing Standards (Normas de Auditoria Gubernamental de Nicaragua) OLACEFS Latin America and the Caribbean Organization of SAIs (Organización Latinoamericana y del Caribe de Entidades Fiscalizadoras Superiores) PDO Project Development Objective SAI Supreme Audit Institution SICA System of Audits Control (Sistema Integrado de Control de Auditorias) UAA Administrative Support Unit of the CGR WBG World Bank Group TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME ...................................................................................................................... 9 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 13 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 13 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 15 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 16 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 21 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P161359 Improving Quality and Efficiency of Public Sector Audit Country Financing Instrument Nicaragua Investment Project Financing Original EA Category Revised EA Category Organizations Borrower Implementing Agency Republic of Nicaragua Nicaragua Comptrollers' General Office Project Development Objective (PDO) Original PDO The project development objective is to improve the quality and efficiency of public sector audit, thereby strengthening external oversight over the management of public resources in Nicaragua. PDO as stated in Legal Agreement (if different from Project Paper) The objective of the Project is to improve the quality and efficiency of public sector audit. Page 1 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A4518 497,000 454,599 454,599 Total 497,000 454,599 454,599 Total Project Cost 497,000 454,599 454,599 KEY DATES Approval Effectiveness Original Closing Actual Closing 18-Apr-2017 01-Jun-2017 30-Jun-2018 28-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 13-Jul-2018 0.23 Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Implementation Schedule 03-Apr-2019 0.43 Change in Results Framework KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 08-Dec-2017 Moderately Satisfactory Moderately Satisfactory 0.10 02 18-Jun-2018 Moderately Satisfactory Moderately Satisfactory 0.23 Page 2 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) 03 03-Jun-2019 Moderately Satisfactory Moderately Satisfactory 0.43 ADM STAFF Role At Approval At ICR Regional Vice President: Jorge Familiar Calderon J. Humberto Lopez Country Director: J. Humberto Lopez Yaye Seynabou Sakho Director: Deborah L. Wetzel Robert R. Taliercio Practice Manager: Daniel J. Boyce Xiomara A. Morel Task Team Leader(s): Lourdes Consuelo Linares Loza Lourdes Consuelo Linares Loza ICR Contributing Author: Lourdes Consuelo Linares Loza Page 3 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Nicaragua remained one of the poorest countries of Latin America and the Caribbean region, but recent strong economic growth had contributed to notable poverty reduction. With per capita gross national income of US$1,800, Nicaragua’s annual economic growth has averaged 4.8 percent in the last six years. This growth, together with an increase in expenditures in Government priority areas, had resulted in a significant reduction in overall poverty (defined as people living under U$S1.6 per day), dropping to 42.5 percent by 2009 and further to 26.9 percent by 2014, according to the 2014 Living Standards Measurement Study. Meanwhile, in the same period, extreme poverty dropped 6 percent, from 14.6 percent in 2009 to 8.3 percent in 2014. Despite progress made, about 1.7 million Nicaraguans (a third of the population) still lived below the overall official poverty line in 2014, with the majority concentrated in rural areas and remote communities with constrained access to basic services. 2. Within that context, the Government recognized the importance of improving governance and accountability of the public sector to accelerate and broaden gains from economic growth. In particular, the Government’s National Plan for Human Development 2012-2016 focused on increasing poverty-related spending and boosting investment in social sectors and rural infrastructure by building on proven models to further expand access and improve efficiency in the delivery of basic services. The Plan emphasized, among others, that establishing credible and effective accountability systems in the public sector would lead to better linkages between spending and outcomes allowing for a more transparent evaluation of public programs, and in turn, improving the quality, effectiveness and efficiency of public sector spending. Beyond the progress made in the area of budget openness and public financial management, Nicaragua still had significant weaknesses in spending oversight both for central and municipal governments, as well as in public sector entities. Building capacity in the country’s oversight institutions so as to improve transparency and accountability was, therefore, essential. 3. The Office of the Comptroller General (CGR) is Nicaragua’s Supreme Audit Institution (SAI) and it is firmly committed to the transparency and accountability agenda in the country. One of the key objectives of the 2011- 2015 CGR’s Strategic Plan for Institutional Development -in force at the time of project preparation- was to increase the number of public institutions subject to oversight that are actually audited. Acknowledging the need for a more comprehensive approach to capacity building, the CGR underwent an external evaluation in 2015, with the support of the Inter-American Development Bank (IDB) and the International Organisation of SAIs, using the SAI Performance Measurement Framework methodology. 4. The evaluation report identified a number of important capacity issues of the CGR. In particular, the report highlighted the lack of updated auditing standards, qualified staff, and comprehensive systems and tools to support the overall audit process, negatively affecting the quality and usefulness of audit work. Audit work was carried out by hand with no automated tools, requiring the use of excessive human and economic resources that are not available, significantly limiting the number of audits in the public sector. This was further exacerbated by the CGR’s inadequate focus on compliance issues, with it only reviewing budget execution reports instead of financial statements, which provide more comprehensive information about the financial situation of an entity or the Government. To improve its performance and respond to the findings of the evaluation report, the CGR, with IDB´s support, prepared a comprehensive action plan, but decided to prioritize the area related to the updating and effective implementation of high-quality professional auditing standards. Page 4 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) 5. The Government auditing standards and the related audit manual were revised in line with the International Standards for Supreme Audit Institutions (ISSAIs) and approved by the CGR on March 23, 2017 . To operationalize the use of the revised auditing standards and audit manual, the CGR developed an awareness on-line training module for CGR staff and internal auditors of public sector entities. However, building the auditors’ skills to effectively apply the new standards and carry out financial, performance and compliance audits in accordance with ISSAIs required a more structured training approach. In addition to providing training on the updated standards, the CGR also needed to put in place an automated tool to support the audit process that would help them ensure compliance with the updated auditing standards, and overall increase the efficiency and effectiveness of the CGR. 6. Building on the efforts initiated by the CGR, partly supported by the IDB, and on the opportunities for knowledge exchange with other SAIs , the CGR prepared a proposal to request World Bank support in three main areas: training of CGR’s auditors and staff from the internal audit units, implementation of an integrated government audit system; and implementation of a quality control system. The proposal was approved to be funded under the Supreme Audit Institution Capacity Development Multi-Donor Trust Fund administered by the World Bank. 7. The World Bank Group (WBG) has been an important ally in the strengthening of public finance management (PFM) in Nicaragua. The Project was fully aligned with the World Bank’s Country Partnership Strategy (CPF) for Nicaragua for FY2013-2017 Report No: 69231-NI. The Strategy identified transparency and accountability in the public sector as a cross-cutting area that was crucial for building broader consensus around development strategies and ensuring adequate oversight of public spending. The Project is expected to strengthen the control environment for effective utilization of Government’s financial resources by empowering the CGR and enhancing professionalism through the use of international auditing standards and modern auditing techniques, among others. Theory of Change 8. Although the project documents did not explicitly include a theory of change as this was not required for World Bank projects at the time of project preparation, the results chain has been constructed from the project activities, outputs and expected outcomes (see Figure 1). The Project’s components and expected outcomes were based on the CGR’s Strategic Plan for Institutional Development and the objective of improved quality and efficiency of Public sector audit was to be achieved through training of auditors working in public sector entities, adoption of a quality control system, as well as the implementation of an automated government audit system. These systems will then be tested through pilot audits which can be mainstreamed by CGR. As a result of these interventions, staff will be better trained and equipped to carry out more efficiently, high-quality audits in accordance with international standards. Component 4, which focuses on project management, monitoring and audit, was expected to facilitate the achievement of the PDO and the intermediate results indicators (IRIs). Therefore, it was not included in the project’s Results Framework. Page 5 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) Figure 1. Project Results Chain Components Activities Outputs PDO/Outcomes Long-term Outcome CGR’s auditors • Carry out train the trainers are trained on 1.Trainin course on ISSAIs updated g of • Deploy on-line and face-to-face auditing Improve the quality of CGR’s training on updated standards. standards and public sector audit auditors • Provide specialized training on audit manual • Quality of Public financial, compliance and performance audit. sector audit is Quality control improved as CGR’s system in auditors are better • Design quality control system accordance trained in carrying 3. based on ISSAIs. Quality with ISSAIs out high-quality • Update CGR’s Code of Ethics implemented audits in accordance Control • Pilot use of the Quality Control System with international System Country’s oversight of * standards. public spending is Government strengthened. audit system (SICA) is • Adapt and implement an Improve the efficiency of operational 2. Gov. integrated government audit public sector audit system (SICA). and used in Audit CGR’s audits • Efficiency of public System • Train audit supervisors and staff sector audit is (SICA) responsible for audit work on Reduction in improved by the use of the use of the system. • Carry out two pilot audits using the time used an automated tool that SICA to carry out supports the audit financial audit process commitments in public sector *Component 3 is presented before Component 2 in the graphic because it complements Component 1 to improve the quality of public sector audit. Project Development Objectives (PDOs) 9. The objective of the Project is to improve the quality and efficiency of public sector audit. Key Expected Outcomes and Outcome Indicators Outcomes Indicators • Improve the quality of public sector audit The Integrated Government Audit System is operational and used in CGR’s audits. Page 6 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) • Improve the efficiency of public sector audit Time to carry out audit commitments is reduced by at least 20 percent. Components 10. The PDO was to be achieved through the following four components identified by the Nicaragua CGR as strategic for its development plan. The activities financed under components 1, 2 and 3 were fully aligned to the CGR’s 2011-2015 Strategic Plan, in force at the time of preparation and directly contributed to some of the strategic objectives reflected in the subsequent CGR’s 2016-2020 Strategic Plan for Institutional Development, specifically objectives 1.1 Expanding the CGR’s audit coverage of public sector entities, 1.2. Expanding the types of audit based on Nicaraguan Government Audit Standards (NAGUN)/ISSAIS, 1.3 Strengthening the audit process with the support of automated tools, and 1.4 Implementing a control and quality assurance system into the audit process. 11. Component 1: Training CGR’s Auditors and Staff from the Internal Audit Units (US$162,575 estimated, US$100,853 actual). This component supported the implementation of the CGR’s training strategy for CGR’s auditors and staff from the internal audit units on the deployment of updated auditing standards and audit manual training. Activities were designed to provide, through a combination of on-line and face-to-face events, i) training on general aspects of the updated standards to a large number of auditors, and ii) specialized training to selected groups of auditors on financial audit, performance audit and compliance audit following the INTOSAI’s Development Initiative - IDI’s 3i Programme. The rationale behind providing specialized training to selected auditors was two-fold; first, to create a group of experts that lead, guide and mentor audit teams in the field, and second to replicate the trainings on a periodic basis. 12. Project design provided for the specialized training to be facilitated through a knowledge exchange arrangement with the Mexican SAI (Superior Audit of the Federation, Auditoria Superior de la Federacion -ASF) as part of the existing OLACEFS’ Internship Program on Performance Audit. 13. Component 2: Implementing an Integrated Government Audit System (US$251,745 estimated, US$302,805 actual). This component supported the adaptation and implementation of an integrated government audit system (SICA for its name in Spanish). The system was to be facilitated by the Chile SAI through a Cooperation Agreement between the Nicaragua CGR and Chile SAI which was signed in June 2017. Chile would also provide technical assistance through its audit and Information and Technology (IT) staff who will guide the adaptation and installation process in Nicaragua, and the Project finance related travel costs. Activities to be financed under the project included: i) the review of the SICA’s features and functionalities to identify and define the adjustments needed to respond to CGR’s needs, ii) the adaptation of existing functionalities, and development of new ones; iii) the provision of the IT infrastructure (hardware and software) required for the installation, administration and operation of the SICA in the CGR; iv) the training of CGR’s IT staff to undertake the adaptation and support system operation and maintenance in the future; and, v) the training of audit staff for the use of the system. 14. The adaptation and implementation processes would be undertaken by a joint CGR team comprising senior staff from the IT and External Audit directorates, who would work under the guidance of the Chile SAI’s team. The system would support the planning, execution, monitoring and supervision of audit engagements. It will help to follow a standard audit process and methodology, facilitate the recording and maintenance of the supporting Page 7 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) documentation, and enforce compliance with required quality control procedures throughout the audit process. The adapted system would be used in two pilot audits to central government agencies. The results of the pilots would then be used to further fine-tune certain functionalities, as well as to attend some additional requirements. 15. Component 3: Implementing a Quality Control System (US$42,680 estimated, US$46,266 actual). This component supported the design and implementation of a quality control system in compliance with ISSAIs that would contribute to delivering high-quality audits. Activities financed under this component included: i) the update of the Code of Ethics, and related documents; ii) the development of the Quality Control Guidelines in accordance with ISSAI No. 40, and iii) provision of training to CGR audit staff in the application of the said guidelines. 16. Component 4: Project Management, Monitoring, and Audit (US$40,000 estimated, US$4,600 actual). This component was expected to support project management related activities, including the partial financing of a Procurement Specialist or a Project Coordinator, operating costs related to Project monitoring, and a single financial audit for the life of the Project. Significant changes during implementation 17. The PDO and outcome targets were not revised throughout the life of the project, but two restructurings were approved by the Bank. The first restructuring processed in June 20181 extended the closing date for one year and approved the financing of additional activities under existing components. The extension of the closing date was needed because the implementation time had been affected by the delays in the approval and signing of the Grant Agreement and delays in the coordination with partner SAIs for key project activities. Thus, the completion of two important activities -implementation of SICA and specialized training- required more time. While the Government of Nicaragua requested a seven-month extension, which from a technical perspective would have been considered adequate, due to the country’s social and political unrest faced in 2018 which affected the operation of the public sector, including the CGR and project implementation, the Bank decided to grant an extension of twelve months until June 28, 2019. 18. The additional activities included under the first restructuring were all designed to further contribute to the achievement of the development objective. Additional activities included the development of guidelines and provision of training on the application of audit sampling standards under Component 1, the development of additional functionalities in the SICA system to carry out of the audit of general-purpose financial statements in compliance with ISSAIs under Component 2; and carrying out of a sensitization and change management workshop for CGR staff to support the transition towards a new operational framework of the CGR. Additional activities were financed out of savings resulting from a simpler methodology for the adaptation of the SICA system, reduction in publication and travel costs, and CGR’s decision to finance 100 percent of the administrative costs. 19. Second restructuring processed in April 2019 was required to adjust the implementation modality for the provision of specialized training under Component 1. This activity was expected to be facilitated by the Mexico SAI (Auditoría Superior de la Federación, ASF) in the framework of a Technical Cooperation Agreement signed between the ASF and the CGR in October 2017. Despite the preparatory work undertaken for this activity, external factors, outside the control of the GoN (management changes and internal reorganization of the ASF), prevented the ASF from 1The restructuring package was processed and approved on June 29, 2018, but because of a system glitch the date in the WB Portal is July 13, 2018. Page 8 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) delivering on their commitment. Since the collaboration with the Mexican ASF was explicitly mentioned in the Grant Agreement, a second restructuring to reflect the change in the implementation modality of the training had to be processed. The amendment to the Grant Agreement allowed the CGR to engage external consultants to carry out the training. This change in the modality of implementation did not affect the quality of the training because the program training itself is already developed as part of the INTOSAI’s Development Initiative’s Program 3i2, and the consultants mainly played a facilitating role. 20. Changes to the Results Framework did not affect the indicator targets, but rather the end target dates for completion. As part of the first restructuring, the end target dates for the PDO Indicators and the intermediate indicators were aligned with the revised project closing date. In addition, a typo error in the target of Intermediate Result Indicator 3 was corrected to 1,000 to ensure consistency with the Project Paper and the original proposal prepared by the CGR. Due to the delays and changes in the implementation modality of the specialized training, the end target date for Intermediate Result 1 Core Team of CGR auditors specialized in Performance audit and Intermediate Result 5 Core team of CGR auditors specialized in ISSAIs was further revised to June 14, 2019 as part of the second restructuring. II. OUTCOME Assessment of Achievement of Each Objective/Outcome 21. The PDO was consistent with the World Bank’s FY2013-2017 Country Partnership Strategy for Nicaragua and continued to be relevant to the CPF for Nicaragua at project closing for the period FY2018-2022, as well as to the CGR’s FY16-20 Strategic Plan for Institutional Development. Objective 7 Improved Data Availability and Public Sector Management Capacity under Pillar 3 of the CPF aims at supporting the GoN to improve fiscal transparency and the effective allocation of public resources. Building on the progress achieved in modernizing public financial management in Nicaragua, the CPF proposes to explore additional opportunities for further enhancing fiscal transparency and the adoption of open data principles. One of the aspects identified in the CPF to support this area is Financial Audit closely related to the improving the quality and efficiency of the CGR. Project activities contributed to: i) enhance CGR’s staff skills for the application of NAGUN and ISSAIs, ii) implement an audit system, and iii) put in place a quality control system in accordance with international standards. The combination of well-trained staff, an automated tool, and a quality control system to strengthen compliance allowed the CGR to ensure the quality of the public sector audit, but also to deliver on its mandate in a more efficient way. The achievement of the two outcomes of the PDO are assessed as follows: 2The Program 3i was created to support SAIs in the implementation of international auditing standards and provides diagnostic tools, handbooks, training material and certified facilitators, which are available to SAIs through their regional bodies, such as the Latin America and the Caribbean Organization of SAIs (OLACEFS) Page 9 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) Outcome 1: To improve the quality of public sector audit 22. The quality of public sector audit has been improved under the Project through two key interventions, i) the training of CGR’s auditors and staff from internal audit units on the updated auditing standards and audit manual, and ii) the implementation of a quality control system, both in accordance with international standards. As the CGR started using the updated NAGUN in 2018, the training strategy supported under the Project covered general aspects of ISSAIs for all audit levels, and specialized training for selected groups of auditors. Knowledge acquired and enhanced skills are already being used in daily work contributing to the effective implementation of the international standards. The CGR now has a group of experts that lead, guide and mentor audit teams in the field, and who are also able to replicate the trainings on a periodic basis. All trainings were facilitated by international consultants with vast experience in the implementation of ISSAIs and in public sector audit (e.g. trainers certified by IDI under Program 3.i for the specialized training). CGR’s training prerequisites, including participation, evaluation of participants, and tests were followed. Specific outputs that contributed to this outcome include: • CGR’s auditors and staff from internal audit units of public sector entities (total 1,214 auditors, above the target of 1,000) have been trained on the general aspects of ISSAIs and updated NAGUN through a combination of on- line (14 events) and face-to-face (6) sessions. As part of this group, 30 auditors became trainers who also supported the deployment of the general training. • Selected group of CGR’s auditors have received specialized training in financial audit (30 auditors), performance audit (20 auditors), and compliance audit (30 auditors) following the IDI’s 3i Programme. • CGR’s auditors (139 auditors) and internal auditors from public sector entities (34 auditors) have been trained on the application of audit sampling standards and related guidelines prepared under the Project. Out of this group, 20 CGR’s auditors have also been trained as trainers for future events. 23. The Quality Control system implemented with Project support is designed to ensure that the CGR staff comply with professional standards as well as legal or regulatory requirements, throughout the audit process, including the issuance of high-quality reports. As part of the quality control system, the Project supported the update of the Code of Ethics and the preparation of the Quality Control and Assurance Guidelines in accordance with ISSAI 40. Both outputs contribute to enhance the quality as follows: • The Code of Ethics, and related documents were approved and are in effect since March 2018. Nearly 600 public officials from the CGR and Internal Audit units of public sector entities participated in face-to-face and virtual awareness events. The updated Code reinforces the relevant ethic requirements of integrity, independence, objectivity and competence and provides guidance, processes and procedures, and instruments to promote an ethical behavior throughout the audit work (e.g. written declarations from staff, handling of breaches of ethical requirements, mechanism to secure independence and staff, etc). • The Quality Control and Assurance Guidelines were prepared and used in two pilot audits. Quality control and the related quality assurance system are an integral part of the international standards (ISSAIs), and an integral part of the updated CGR audit standards (NAGUN). The guidelines prepared under the Project constitute the tool to operationalize those requirements by i) assigning responsibilities for quality control throughout the audit process and responsibilities for the quality assurance function; ii) establishing quality control procedures; and iii) providing forms to document different quality control activities. CGR’s audit staff were trained in the application of the Page 10 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) guidelines, and the Quality Control Unit on the assurance function. The pilots allowed the teams to use the guidelines in two audit engagements with the guidance of the consultant, so they received additional hands-on- training on the application of the procedures and requirements included in the guidelines. Twenty-one auditors assigned to supervisory and/or quality control functions were trained in the application of the guidelines. Following the Project closing, the CGR filled in the two vacant positions in the Quality Control Unit that is in charge of monitoring the operation of the quality control system. Outcome 2: To improve the efficiency of public sector audit 24. Government audit system (SICA) is operational and used in CGR’s audits with demonstrated increased efficiency. The SICA system has been adapted in all core aspects, and the IT infrastructure (hardware and software) required for the operation and administration of the system has been installed and is fully operational. The system is also installed in the laptops used by the audit staff in the field work. Each new audit engagement created in the system allows the audit team to document the review and complete mandatory steps in the system as the review is carried out. Audit teams can work off-line -if needed- and as soon as they are connected to the CGR network, all updates are uploaded into the server. The system has been used in two pilot audits -Ministry of Finance and Public Credit (MHCP) and Ministry of Development, Industry and Trade (MIFIC)- and the time used in each of those audits -when compared to the time used in the previous year (without the system) shows a 20% reduction of time (from 60 days to 48 days)3. The feedback received from the pilot audits will be used for further improvements. 25. An integrated CGR team comprising staff from the IT and External Audit Directorates has been trained in the design, operation and administration of the system. Going forward this team will lead the roll out of the system, will provide training, and support any additional improvements. CGR’s auditors (139) have been trained on the system’s features, and main operational aspects. Hands-on-training will continue as the system is used in new audits. CGR’s decision is to gradually expand the use of the system to additional pilot audits of larger and more complex entities throughout CY2019. Accordingly, SICA was used in additional five audits in CY2019 -after project closing- and is planned to be used in twelve audits in 2020. The CGR’s Council approved the use of SICA in accordance with the Annual Audit Plan on March 10, 2020. 26. The activities supported through the Project, mainly the implementation of the SICA system, are to a certain extent transforming the way public sector audit is carried out, but project design did not provide for a change management component/sub-component, neither the time to properly accompany the transition. The pilot audits carried out under the Project revealed that the use of the system throughout the audit process represents a major change in the way the audit teams interact with the audited entities, document the process, carry out internal supervision and quality control, manage the time and staff assignment, etc. Young auditors are more easily embracing the changes, and clearly committed to them, but it might be more challenging for senior staff, for whom the use of an automated tool can be challenging and disrupting. A sensitization and change management workshop was carried out before the project closing date. The main purpose of the workshop was to evaluate project main results, allow audit teams that participated in the pilots to share their experience, and reflect on some of the challenges moving forward. Mindful of those challenges, the CGR carried out a change has decided to do a gradual roll-out of the system, adjusting the configuration of the audit teams to foster collaboration between senior staff and young auditors, and 3 The base line information on the number of days used to complete an audit was obtained from the time recording system maintained by the CGR for every audit engagement. Time is broken down by audit phase -planning, execution and issuance of the report. Page 11 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) complementing the training sessions with hands-on and on the job training. The purpose behind this strategy is to strengthen and secure the ownership of the audit staff at all different levels for a more effective implementation. Consequently, the CGR decided to put off the formal approval of the SICA system by the CGR’s Council, which was originally expected to happen before project closing. Overall Outcome Rating 27. The project’s overall outcome is rated Moderately Satisfactory. The Project’s objective is relevant to the current CPF for Nicaragua as it contributes to the improving public sector management capacity objective of the CPF, and it is also relevant for CGR’s Strategic Plan. Project objectives have been achieved with moderate shortcomings, with exception of the CGR’s decision to postpone the Council’s formal approval of the SICA system, which happened in March 2020 (after project closing). The project objectives are expected to be fully achieved as audit staff are getting more familiar with the system through the additional pilot audits and with the formal approval of the use of SICA in accordance with the CGR’s Annual Audit Plan. SICA is operational and its use compels the audit teams to follow an orderly process from planning through execution and issuance of audit reports, all in accordance with international standards in auditing. The system provides audit procedures and requires that supervision and quality control throughout the audit process is not only documented, but more important that it happens on a timely basis. Its use in the two pilot audits has demonstrated that it would allow CGR to carry out quality audits in a more efficient way. 28. With project support CGR’s auditors have been trained on the updated NAGUN and ISSAIs to carry out high- quality audits, and a quality control system to enforce compliance with the new standards is now in place. The combination of general and specialized training of selected auditors will allow the CGR to continuing reinforcing staff skills, not only replicating formal training, but also through on-the-job training and more punctual guidance provided by the teams specialized in different types of audits. 29. The combination of well-trained staff, an automated tool and a quality control system, all in accordance with international standards will lead to better quality and more efficient public sector audit. To this end, it will be important that CGR Council, and donor partners accompany and support the consolidation of the changes introduced by the project, for instance by promoting twining arrangements for the first performance audits and supporting the formal assessment of the use of SICA after the first year of operation. 30. Project performance was rated Moderately Satisfactory throughout the life of the project. While the PDO remained relevant to the country and WB strategy, moderate shortcomings affected implementation progress at different stages. Those shortcomings were related for instance to delays in the coordination with partner SAIs, but also to the likelihood that some factors outside the control of the CGR put at risk the completion of key activities and achievement of the Project objectives. Actions taken by the CGR and by the Bank (e.g. extension of the closing date, changing the modality of implementation of the specialized training) helped the Project to deliver on most of the agreed results. Other Outcomes and Impacts 31. The implementation of SICA in Nicaragua constitutes a good example of coordination, collaboration and complementarity of efforts from different stakeholders to achieve concrete results that can be replicated in other interventions in the region. While the Chile CGR facilitated the system and the technical expertise of its staff, the adaptation and implementation would not have been possible without the project financing the required IT Page 12 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) infrastructure (hardware and software), training of the CGR’s IT staff and travel costs of Chilean team to support and guide the adaptation and installation process in the field. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 32. CGR’s commitment and ownership was critical to achieve project objectives. Project outcomes were translated into specific objectives under the CGR’s strategic plan (FY18-20) with well-defined targets for the coming years. The latter reflects the CGR’s commitment not only to the Project, but to the effective implementation and sustainability of the improvements supported under the grant. Project implementation relied fully on CGR’s technical units, instead of external consultants. Implementation of component 2, for instance, was supported by a team comprising senior staff from the External Audit Directorate and IT Directorate who were fully dedicated to the Project. Additionally, the CGR decided to finance 100 percent of the administrative staff required for project implementation. Institutional and management commitment continued towards project closing, despite changes in the President of the CGR’s Council in early 2019. 33. Project design and implementation relied significantly on knowledge exchange and collaboration with other SAIs in the Region, and the benefits of that approach significantly surpassed the challenges. Partnering with other SAIs, at least based on the experience with Chile, proved to be an effective and efficient mechanism for capacity building. The CGR was able to interact with peers who are operating within the public sector, who understand the challenges faced by SAIs and who can provide practical advice to address different issues. However, there were also challenges in concretizing, mobilizing and operationalizing the partnerships. It took significant effort and time to coordinate the activities and required flexibility on both sides to accommodate availability of staff and reschedule some activities and field visits. The close interaction between the CGR and Bank team allowed timely assessment of the impact of those delays on project implementation and to proactively seek solutions, either by extending the closing date or adjusting the implementation modality of a key activity. 34. Despite the initial coordination delays, the collaboration with the Chile SAI worked well and significantly contributed to achieve the project PDO. The Chile CGR not only facilitated the SICA system for its use in Nicaragua, but also provided technical assistance through its audit and IT staff. The Chilean team trained the CGR’s team in the design and operation of the system, guided the adaptation process, supported the installation of the IT infrastructure and collaborated in the training of CGR’s auditors. That interaction also fostered discussions about broader aspects related to the operation of an SAI, such as institutional planning, risk-based approach, engagement with the citizens, etc.; which ideally may lead the CGR to pursue additional reforms and improvements. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME Bank performance 35. Overall, WB performance is considered moderately satisfactory. The World Bank team initiated the dialogue with the CGR -Council and management team- following the completion of the CGR’s independent assessment using the SAI Performance Measurement Framework, and the CGR’s request for support in the implementation of its strengthening action plan. Preparation process included discussions with CGR’s management and technical teams as well as key donors, such as the IDB to secure harmonization and avoid duplication. As a result, a proposal for funding Page 13 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) under the SAI CDF Grant (administered by the WB) was presented. Three key aspects considered in assessing the project during preparation were: i) CGR’s leadership and ownership evidenced by having a project proposal fully aligned with the its Strategic Plan for Institutional Development, ii) alignment to WB CPF objectives, and iii) a focus on delivering results that demonstrate improvements in SAI performance in delivering high-quality audits. The last aspect led to focusing the project objective and scope to only one area of the CGR’s plan, the quality and efficiency of public sector audit by supporting the effective implementation of ISSAIs through three interventions: enhancing staff skills for the application of the new standards, putting in place an automated tool that supports the audit process and putting in place a quality control and assurance system. 36. Implementation Status Reports were published regularly, and the World Bank did draw attention to implementation challenges in ISRs, and required action was taken. As part of World Bank implementation support, the team carried out two field supervision missions, and also held periodic follow-up virtual meetings to review progress, provide guidance on specific technical/fiduciary issues, and discuss alternatives to address implementation issues, such as the delays in the specialized training. Continuing interaction between the WB team and the CGR also allowed to timely identify the need for restructuring to extend the project closing date and later change the implementation modality for the specialized training. Aide-Memoirs and ISRs flagged implementation challenges, delays, and risks, including those that were not under the CGR’s control. Support with financial management and procurement aspects was also provided on a regular basis. Compliance issues 37. Overall, CGR complied with the Grant Agreement, including fiduciary requirements, although continuing support from technical and fiduciary teams was required. The Procurement Plan was approved during project preparation and it did not include complex procurement. The Bank team provided guidance for the preparation of terms of reference, technical specifications, and bidding documents, and CGR’s team participated in different training events and workshops. Procurement plans were timely updated in the Systematic Tracking of Exchanges in Procurement (STEP) system, but the uploading and recording of procurement process remained challenging throughout project life. As it relates to financial management, the CGR complied with the timely submission of the semi-annual Interim Financial Reports, and single audit report as of project closing. However, weaknesses in the accounting records and documentation of expenditures were identified during project supervision; leading to a downgrade in the FM ISR rating. A corrective action plan was agreed, and the CGR was able to overcome the shortcomings. From the Safeguards side, the Project was rated as Category C because it did not trigger any environmental safeguards or social policies as no negative impacts were anticipated as a result of this project. Risk to development outcome 38. The risk to development outcome is modest. In future, it is unlikely that the results achieved by the Project will be reversed mainly because they respond to the requirements included in the updated audit standards (NAGUN) based on ISSAIs which were formally adopted and are in use since 2017. Accordingly, the reforms supported by the Project are part of the CGR’s Strategic Plan for Institutional Development. Targets are clearly defined in the plan and the President of the CGR’s Council and Management team are responsible and accountable for the achievement of those targets. Additionally, the instruments and guidance prepared under the Project are formally approved by the Council and are mandatory for CGR staff. Thus, changes supported by the Project in terms of system, staff skills and Page 14 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) quality control will be maintained and gradually expanded, mainly in terms of the system. However, the CGR will need to continue making every effort for the retention of trained staff in all areas. V. LESSONS LEARNED AND RECOMMENDATIONS 39. Technical cooperation among SAIs can be an effective and efficient way to support modernization and capacity building, but challenges and risks to the operationalization of those arrangement need to be assessed early in a project. Knowledge sharing platforms are active among SAIs in the region, but still there is need to officialize those arrangements through MoUs or other type of agreements, have clarity on the activities needed, secure funding to mobilize the cooperation and work on an agreed upon plan that takes into account staff availability and other constraints. Technical cooperation may also be affected by external factors -political, institutional- completely outside the control of the receiving entity, which may cause implementation delays. Mitigating those risks may sometimes require adapting or redesigning the interventions and doing so in a timely manner is important. 40. Implementation of IT solutions can be transformational, but also disruptive to the point it affects the regular operation of an agency if not supported by a change management strategy. Even in the case of modest or not very complex IT solutions (like the SICA), there is need to appreciate also the non-technical aspects of the envisioned change, such as the effects on the daily work of the users and the possible ways to support the transition. 41. Capacity building interventions should also consider and provide for the time needed for the reforms to be assimilated by the implementing entities. Project design and implementation plans tend to focus more on the timely delivery of outputs. In addition to that, reflecting on the time and the support needed for the institutions to understand, develop ownership and finally embrace the changes is equally important and should be factored in the . project design, even when financed by small trust funds. Page 15 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS • Outcome 1: Improve the quality of public sector audit. • Outcome 2: Improve the efficiency of public sector audit. A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Quality and efficiency of public sector audit is improved. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Integrated Government Audit Yes/No N Y Y N System (SIAG) implemented and formally adopted for use in 07-Nov-2016 30-May-2018 26-Jun-2019 26-Jun-2019 CGR's audits Comments (achievements against targets): The integrated government audit system has been adapted and installed to be used in Nicaragua, and it has been used in the two pilot audits under the project and additional audits in 2019 and 2020.. Formal adoption by the CGR's council though has been delayed Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion 20% Reduction in the time Days 60.00 48.00 48.00 48.00 Page 16 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) (number of days) used to carry 07-Nov-2016 30-May-2018 26-Jun-2019 26-Jun-2019 out financial audit commitments in public sector entities Comments (achievements against targets): A comparison of the number of days used to carry our the audits for the two pilot entities without the system (2017 audit) and with the system (2018 audit) show a reduction in the number of days. A.2 Intermediate Results Indicators Component: Training of CGR's auditors and staff from the internal audit units Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Core team of CGR auditors Number 0.00 20.00 20.00 20.00 specialized in Performance audit 07-Nov-2016 30-Apr-2018 21-Jun-2019 21-Jun-2019 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Core team of CGR auditors Number 0.00 30.00 30.00 30.00 Page 17 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) specialized in ISSAIs 07-Nov-2016 30-Apr-2018 21-Jun-2019 21-Jun-2019 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion CGR’s auditors and internal Number 0.00 1000.00 1000.00 1214.00 auditors of 249 entities trained in overall use of updated 07-Nov-2016 31-Oct-2017 30-Jan-2018 11-Jun-2018 NAGUN based on ISAISs Comments (achievements against targets): Component: Implementation of an integrated government audit system Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion SIAG adapted, installed and in Yes/No N Y Y Y operation for being used by CGR staff in pilot audits 07-Nov-2016 30-Nov-2017 30-Apr-2019 30-Apr-2019 Comments (achievements against targets): Page 18 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) Component: Implementation of a quality control system Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Quality control system Yes/No N Y Y Y implemented in accordance with ISSAI No. 40 07-Nov-2016 30-Apr-2018 28-Sep-2018 05-Apr-2019 Comments (achievements against targets): Page 19 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 Improve the quality of public sector audit 1. Integrated Government Audit System (SIAG) implemented and formally Outcome Indicators adopted for use in CGR's audits 1. CGR’s auditors and internal auditors of 249 entities trained in overall use of updated NAGUN based on ISAISs Intermediate Results Indicators 2. Core team of CGR auditors specialized in ISSAIs 3. Quality control system implemented in accordance with ISSAI No. 40 4. Core team of CGR auditors specialized in Performance audit Key Outputs by Component 1. CGR’s auditors are trained on updated auditing standards and audit manual. (linked to the achievement of the Objective/Outcome 1) 2. Quality control system in accordance with ISSAIs implemented. Objective/Outcome 2: Improve the quality and efficiency of public sector audit 1. 20% Reduction in the time (number of days) used to carry out financial audit Outcome Indicators commitments in public sector entities 1. The integrated government audit system (SICA) adapted, installed and in Intermediate Results Indicators operation for being used by CGR staff in pilot audits. Key Outputs by Component 1. Government audit system (SICA) is operational and used in CGR’s audits. (linked to the achievement of the Objective/Outcome 2) Page 20 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Training CGR’s auditors and staff from the Internal Audit 0.16 .0.10 .62 Units. Implementing an Integrated Governmental Audit System 0.25 0.30 1.20 (SIAG) Implementing a Quality 0.04 .05 1.25 ControlSystem . Project management, 0.04 0.004 0 Monitoting and audit Total 0.49 0.45 0.92 Page 21 of 22 The World Bank Improving Quality and Efficiency of Public Sector Audit (P161359) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS The Implementation Completion and Results Report (ICR) was shared with the CGR in December 2019 for comments. CGR representative confirmed their agreement with the content and views expressed in different sections on January 13, 2020. No further comments were received. Page 22 of 22