GUATEMALA Enhanced Fiscal and Financial Management for Greater Opportunities DPL Series Report No. 141817 NOVEMBER 14, 2019 © 2019 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2019. Guatemala—Enhanced accuracy of the data included in this work. The Any queries on rights and licenses, including Fiscal and Financial Management for Greater boundaries, colors, denominations, and other subsidiary rights, should be addressed to Opportunities DPL Series. Independent information shown on any map in this work do World Bank Publications, The World Bank Evaluation Group, Project Performance not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC Assessment Report 141817. Washington, DC: World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: World Bank. territory or the endorsement or acceptance of pubrights@worldbank.org. such boundaries. Report No.: 141817 PROJECT PERFORMANCE ASSESSMENT REPORT GUATEMALA Fiscal Space for Greater Opportunities: First Programmatic DPL (P131763) (IBRD-82030) and Enhanced Fiscal and Financial Management for Greater Opportunities: Second Programmatic DPL (P133738) (IBRD-83850) November 14, 2019 Human Development and Economic Management Independent Evaluation Group Currency Equivalents (annual averages) Currency Unit = Guatemalan Quetzal (GTQ) 2015 $1.00 7.76 2016 $1.00 7.60 2017 $1.00 7.35 2018 $1.00 7.52 Abbreviations CES Consejo Económico Social (Economic and Social Council) DPL development policy loan GDP gross domestic product IDB Inter-American Development Bank IMF International Monetary Fund M&E monitoring and evaluation PPAR Project Performance Assessment Report SAT Superintendencia de Administración Tributaria (tax administration service) All dollar amounts are U.S. dollars unless otherwise indicated. Fiscal Year Government: January 1 – December 31 Director-General, Independent Evaluation Ms. Alison Evans Director, Strategy and Operations Ms. Sophie Sirtaine Manager, Country Programs and Economic Management Mr. Jeff Chelsky Task Manager Mr. Željko Bogetić ii Contents Preface ................................................................................................................................................................. vi Summary ............................................................................................................................................................vii 1. Background and Context ..........................................................................................................................1 2. Relevance of the Objectives and Design.............................................................................................5 Objectives .......................................................................................................................................................5 Design ..............................................................................................................................................................5 3. Achievement of the Objectives ...............................................................................................................7 Strengthening Tax Administration and Tax Policy ............................................................................ 7 Strengthening Budget Management and Increasing Results Orientation of Public Spending....................................................................................................................................................... 10 Improving the Management and Coordination of Social Policies ..............................................11 4. Outcome....................................................................................................................................................... 12 Risk to Development Outcome ............................................................................................................. 13 Bank Performance ..................................................................................................................................... 13 Quality at Entry ......................................................................................................................................................... 13 World Bank Supervision ......................................................................................................................................... 14 Borrower Performance ............................................................................................................................. 14 Government Performance..................................................................................................................................... 14 Monitoring and Evaluation ..................................................................................................................... 15 Design .......................................................................................................................................................................... 15 Implementation and Use ....................................................................................................................................... 15 5. Lessons.......................................................................................................................................................... 15 Bibliography..................................................................................................................................................... 18 Box Box 3.1. What Works: Using Behavioral Insights and “Nudges” to Improve Tax Compliance ..8 Tables Table 1.1. Guatemala Select Economic Indicators, 2011–18...................................................................3 iii Table 1.2. Development Policy Loan Projections versus Outturn of Select Economic Indicators, 2012–18 .......................................................................................................................4 Table 3.1. Guatemala DPL Series: Achievement of Target Indicators under the First Two Objectives........................................................................................................................................9 Table 3.2. Guatemala DPL Series: Achievement of Target Indicators under the Third Objective ........................................................................................................................................11 Appendixes Appendix A. Basic Data Sheet ................................................................................................................... 23 Appendix B. Development Policy Loan Policy and Results Matrixes ......................................... 28 Appendix C. List of Persons Met .............................................................................................................. 33 Appendix D. Approach, Methods, and Ratings .................................................................................. 34 This report was prepared by Željko Bogetić (task team leader), with research assistance support from Jorge Coj, Amshika Amar, and Johan Lopez, who assessed the program during May–August 2019. The report was peer-reviewed by Konstantin Atanesyan and panel- reviewed by Robert Lacey. Carla Fabiola Cole and Dung Thi Kim Chu provided administrative support. iv Principal Ratings Operation 1: Fiscal Space for Greater Opportunities: First Programmatic Development Policy Loan (P131763), Approved on September 27, 2012 Operation 2: Enhanced Fiscal and Financial Management for Greater Opportunities: Second Programmatic Development Policy Loan (P133738), Approved on June 17, 2014 Indicator ICRa ICRRa PPAR Outcome Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Risk to development Substantial Substantial High outcome Bank performance Satisfactory Moderately satisfactory Moderately unsatisfactory Borrower performance Moderately satisfactory Moderately satisfactory Moderately unsatisfactory Note: ICR = Implementation Completion and Results Report; ICRR = Implementation Completion and Results Report Review; PPAR = Project Performance Assessment Report. a. The ICR is a self-evaluation by the responsible Global Practice. The ICRR is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. Key Staff Responsible Fiscal Space for Greater Opportunities Management Appraisal Completion Project team leader Marco Antonio Hernandez Ore Marco Antonio Hernandez Ore Sector Manager or Practice Manager Pablo Saavedra Pablo Saavedra Sector Director or Senior Global Practice John Panzer John Panzer Director Country Director J. Humberto Lopez J. Humberto Lopez Enhanced Fiscal and Financial Management Management Appraisal Completion Project team leader Marco Antonio Hernandez Ore Marco Antonio Hernandez Ore Sector Manager or Practice Manager Auguste Tano Kouame Pablo Saavedra Sector Director or Senior Global Practice J. Humberto Lopez John Panzer Director Country Director Maryanne Sharp J. Humberto Lopez v Preface This Project Performance Assessment Report (PPAR) evaluates a series of two development policy loans (DPLs) to Guatemala: Fiscal Space for Greater Opportunities ($200 million, P131763), and Enhanced Fiscal and Financial Management for Greater Opportunities ($340 million, P133738). The assessment aims to verify whether the operation achieved its intended outcomes, to understand what worked well and what did not, and to draw lessons for the future. The PPAR was prepared by Željko Bogetić (task team leader) under the supervision of Jeff Chelsky (manager). Preparation benefited from research analyst support from Jorge Coj during the Guatemala mission and team assistance from Jairi Yesenia Hernandez Coro in the Guatemala City office. Research support from Amshika Amar and Johan Lopez and team assistance from Dung Thi Kim Chu and Carla Fabiola in the Washington, DC, office are also gratefully acknowledged. The PPAR team wishes to express sincere gratitude to the government officials, stakeholders, and World Bank staff interviewed, who provided their perspectives and valuable information in the course of this assessment. The first DPL was approved on September 27, 2012, declared effective on November 21, 2013, and closed as planned on December 31, 2013. The second loan was approved on June 17, 2014, declared effective on February 26, 2015, and closed as planned on March 31, 2015 (Appendix A). Delays in effectiveness were due to the long process for obtaining Parliament’s approval of the loans. Both loans were fully disbursed on effectiveness. The World Bank extended the DPL series in support of the government’s program, which included tax reform, the Fiscal Pact, and the Zero Hunger Pact. There was no parallel program with the International Monetary Fund (IMF). The World Bank coordinated with the IMF and Inter-American Development Bank, the main development partners, on budget support–related issues and policy dialogue. This report presents findings based on a review of program documents, Implementation Completion and Results Reports, Implementation Completion and Results Report Reviews, supervision reports, IMF and World Bank reports, and other relevant materials. An Independent Evaluation Group mission visited Guatemala City during April 22–27, 2019, to interview government officials and other stakeholders, including World Bank, IMF, and Inter-American Development Bank staff (see appendix C for the complete list of persons interviewed). World Bank staff members, representatives of development partners, and other information providers were also interviewed at headquarters. Following standard Independent Evaluation Group procedures, the draft PPAR was sent to the borrower for comments; no comments were received. vi Summary The objectives of the series were to (i) strengthen tax administration and tax policy, (ii) strengthen budget management and increase the results orientation of public spending, and (iii) improve the management and coordination of social policies. Relevance of objectives was substantial. They were closely aligned with the World Bank’s strategy and the government’s priorities. They were relevant to country conditions, in particular the need to address the long-standing problems of low domestic revenue mobilization and the adequacy and quality of public spending, especially that directed toward the poor. Relevance of design was modest. Key policy areas were identified through analytical work, especially the Public Expenditure Review, and through policy dialogue. However, the links between policy measures and intended results were not always clear, and the results indicators were not sufficiently outcome oriented. Underlying political economy factors behind the long-standing problem of low government revenues were not addressed. Instead, it was assumed that the technical design and focus on upstream revenue measures would be enough to ensure parliamentary passage of the key tax reforms and their effective implementation. This approach, however, ran against Guatemala’s history of legislative and interest group resistance to tax reform. Achievement of the objectives was modest. Targeted increases in government revenues did not materialize; rather, by the end of 2018, the ratio of revenue to gross domestic product was a full percentage point lower than at the inception of the program in 2012. No data are available to gauge improvements in tax administration. Evidence on results- based budgeting is mixed and suggests that formal adoption had not yet translated into improved results. The creation of the new Ministry of Social Development and the activities of the Economic and Social Council raised awareness of social issues and involved various stakeholders and achievement in this area has been more substantial, but coordination of social policies does not appear to have improved significantly. Reflecting substantial relevance of objectives, modest relevance of design, and modest achievement of objectives, the overall outcome for the series is assessed as moderately unsatisfactory. The reasons for the limited achievement of the tax administration and tax policy objectives lie in the complex and deep-seated forces that prevent government revenues from increasing on a sustainable basis. First, there is a strong ideological opposition to higher government revenues among influential groups in Parliament, the private sector, and the government. Second, low revenues mean that public investment and pro-poor programs are seriously underfunded. This situation, together with widespread corruption, crime, violence, and political scandals, has resulted in low public trust in vii government. Third, an estimated 80 percent of economic activity in Guatemala is in the informal sector, reflecting a culture of noncompliance and lack of trust in government. Finally, the tax administration’s weak capacity has constrained the scope of reform. Bank performance, both at entry and during implementation, was moderately unsatisfactory. Program design did not adequately consider or address the deeply rooted opposition to reform of tax administration and tax policy. Prior actions, and links between objectives and results indicators, were weak. The monitoring and evaluation design had significant shortcomings, such as the absence of any indicator measuring improvement in tax administration. Supervision missions noted inadequate progress but could not resolve issues stemming from design weaknesses. This Project Performance Assessment Report offers the following lessons: • Tax administration and tax policy reforms in the face of major governance issues and long-standing opposition from influential interest groups are unlikely to be successful, even if backed by the World Bank’s analytical support, policy dialogue, and financing. Under these conditions, directly and indirectly targeting the governance issues over a longer period is necessary. • Achieving progress on results budgeting requires strengthening of capacity, political commitment, sound monitoring and evaluation indicators, and cross- agency collaboration. • Achieving results in policy lending requires a sound results framework, a credible theory of change, close linking of objectives with policy actions, and outcome-oriented target indicators. Sophie Sirtaine Director, Strategy and Operations Independent Evaluation Group viii 1. Background and Context 1.1 After the conclusion of the 36-year civil war and the peace accord in 1996, Guatemala’s subsequent governments have attempted to rebuild institutions and the fiscal revenue base, deal with the pressing postconflict issues, and restart broad-based growth conducive to poverty reduction. Guatemala, a lower-middle-income country with a per capita income of $4,060 and a population of 17 million in 2017, is the largest economy in Central America. During the 2011–18 period, despite political tensions and institutional weaknesses, the government maintained low fiscal deficits, public debt, and inflation, reflecting long-standing political aversion to fiscal and monetary instability. 1.2 Growth averaged above 3 percent, but per capita growth was much lower due to a rapid increase in population, even after large-scale emigration. Extreme poverty, measured by the international poverty line of $1.9 per person per day, fell from 11.1 percent of the population in 2006 to 8.7 percent in 2015 (the latest estimate), but poverty measured at the much higher national poverty line shows an unfavorable trend, increasing from 51.0 percent to 59.3 percent. Indigenous peoples have the highest level of poverty. During the same period, the income Gini coefficient declined from 54.6 but remains high at 48.3. The Human Development Index lags that of the Latin America and the Caribbean region. Persistent low government revenues have severely limited financing for basic public services and public investment. Weak institutional quality is reflected in Worldwide Governance Indicators,1 which showed no significant improvement over the past two decades. Crime, violence, and corruption are identified as significant problems for both security and efficient functioning of the public and private sectors (World Bank 2005, 2012c). 1.3 The context for the development policy loan (DPL) series was characterized by a major and long-standing policy and development challenge: a government revenue –to– gross domestic product (GDP) ratio of only 11.6 percent in 2011 (one of the lowest in Latin America), reflecting a restricted tax base and limited nontax sources of revenues (Table 1.1). The World Bank Group’s Systematic Country Diagnostic attributed high and persistent poverty in Guatemala to, among other factors, a weak social safety net, directly attributable to limited availability of financing (Sanchez, Scott, and Lopez 2016). This is of particular concern given households’ high vulnerability to frequent natural disasters. Previous country diagnostics have repeatedly pointed to this issue and its importance for broader economic progress. “The greatest macroeconomic challenge for Guatemala lies in improving its internationally low public revenue–to-GDP ratio. The Peace Accords set a target of 12 percent of GDP to be reached by 2002, a significant increase from the level 1 that existed at the time of the Accords (7.9 percent). Unfortunately, progress has been slower than expected and the dearth of public funds continues to limit investments in key growth-enhancing investments, including those that are critical to boost the pace of economic and social progress. Moreover, with a relatively low level of tax receipts, fiscal policy has few degrees of freedom to accommodate shocks and provide countercyclical stimulus when needed.” (World Bank 2005, iii) 1.4 Why has the revenue-to-GDP ratio not risen in the face of massive social and investment needs?2 Various studies point to interaction among factors, including a low- trust social equilibrium, fragmented social contract, and weak taxpayer culture, where the private sector and households do not trust the government and resist participation in the formal, tax-paying economy. These factors are reinforced by the extremely limited services that the government is able to provide to its citizens, as well as by sizable tax expenditures in favor of select sectors and enterprises (World Bank 2012c, 16). 1.5 In reviewing performance at the time of the first DPL’s approval, the Independent Evaluation Group’s Country Partnership Strategy Completion Report Review for Guatemala drew the following lesson for future Bank Group engagement in Guatemala: “The most important of these [lessons] is the need for realism in the assessment of the prospects for results, with a much sharper focus on the degree of ownership of various stakeholders and the effects this is likely to have on prospects for policy reform and institutional development. The political economy factor overwhelms any of the design and implementation factors under the WBG’s [World Bank Group’s] control.” (World Bank 2012a, 2) 2 Table 1.1. Guatemala Select Economic Indicators, 2011–18 Indicator 2011 2012 2013 2014 2015 2016 2017 2018 Real GDP 4.2 3 3.7 4.2 4.1 3.1 2.8 3.1 CPI (end of period) 6.2 3.8 4.3 3.4 2.4 4.4 4.4 4.4 Fiscal indicators (% of GDP) Budgetary expenditure 14.4 14 13.8 13.4 12.3 12.1 12.1 12.3 Wages and salaries 3.8 3.8 4.0 4.1 4.2 4.0 4.0 4.0 Interest payments 1.4 1.5 1.6 1.4 1.6 1.5 1.4 1.4 Capital expenditures 3.2 2.9 3.0 2.9 2.2 2.1 2.2 2.4 Budgetary revenue 11.6 11.6 11.6 11.5 10.8 11 10.8 10.6 Budget balance −2.8 −2.4 −2.1 −1.9 −1.4 −1.1 −1.3 −1.8 Monetary and external indicators Credit to private sector 28.7 31.7 33.1 33.6 35.2 35.9 37.3 34.3 (% of GDP) Interest (end of period) — — 5.0 4.0 3.0 3.0 2.7 — Current account −3.4 −2.6 −3.2 −2.1 −0.2 1.5 1.1 0.8 balance Foreign direct — — –2.3 –2.2 –1.7 –1.6 –1.3 –1.0 investment Net international — 6,197 6,433 6,587 7,077 7,498 7,498 — reserves ($, millions) Public debt (% of GDP) 23.9 24.6 25.1 24.3 24.2 24.5 24.7 24.5 Short-term debt ($, 2.3 0.9 0.9 0.9 0.8 0.9 1.1 — billions) Short-term debt (% of 13.8 13.0 13.0 10.9 10.1 9.0 — — total reserves) Credit rating (Moody’s) — — — — Ba1 Ba1 Ba1 Ba1 GDP, nominal ($, billions) 47.7 50.4 53.9 58.7 63.8 68.7 74.6 78.9 Sources: Bank of Guatemala; International Monetary Fund (IMF) 2019; IMF DataMapper, https://www.imf.org/external/datamapper/NGDP_RPCH@WEO/OEMDC/ADVEC/WEOWORLD; IMF, Fiscal Monitor; IMF International Financial Statistics database; Moody’s, https://www.moodys.com/credit-ratings/Guatemala-Government-of- credit-rating-600011455; Statistical Institute of Guatemala; World Bank Data Catalog, https://datacatalog.worldbank.org/dataset/international-debt-statistics; World Bank Open Data, https://data.worldbank.org/indicator/DT.DOD.DSTC.IR.ZS?locations=GT&view=chart. Note: — = not available; CPI = consumer price index; DPF = development policy financing; DPL = development policy loan; GDP = gross domestic product. 3 Table 1.2. Development Policy Loan Projections versus Outturn of Select Economic Indicators, 2012–18 2012 2013 2014 2015 2016 2017 2018 Indicator (projected) Real GDP growth (%) At the time of DPL approval (Sept. 2012) 3 3.5 3.5 3.6 3.6 — — As of 2018a 3 3.7 4.2 4.1 4.1 3.1 2.8 Difference (actual minus projected)b 0 0.2 0.7 0.5 −0.5 3.1 2.8 Government revenues (% of GDP) At the time of DPL approval (Sept. 2012) 11.7 11.7 11.9 11.9 11.9 — — As of 2018a 11.6 11.6 11.5 10.8 11.0 10.8 10.9 Difference (actual minus projected)b 0.1 0.1 0.4 1.1 0.9 — — Sources: Bank of Guatemala; International Monetary Fund (IMF) 2019; IMF DataMapper, https://www.imf.org/external/datamapper/NGDP_RPCH@WEO/OEMDC/ADVEC/WEOWORLD; IMF, Fiscal Monitor; IMF International Financial Statistics database; Statistical Institute of Guatemala; World Bank Data Catalog, https://datacatalog.worldbank.org/dataset/international-debt-statistics; World Bank Open Data, https://data.worldbank.org/indicator/DT.DOD.DSTC.IR.ZS?locations=GT&view=chart. Note: The values for 2012 for “As of 2018” are from the Guatemala 2016 IMF Article IV Consultation report (IMF 2016). The values for 2013 and 2014 for “As of 2018 (projected)” are from the Guatemala 2016 IMF Article IV Consultation report (IMF 2016). For the line “At the time of DPL [development policy lending] approval,” the value for 2012 is actual, and values for 2013–17 are projections from the DPL program document. The value for 2018 is the projection from the 2018 IMF Article IV Consultation report (June 8, 2018) (IMF 2018a). — = not available; CPI = consumer price index; DPL = development policy loan; GDP = gross domestic product. a. Actual values for years 2012–17 and projection for 2018 are from the Guatemala Article IV Consultations for 2016 and 2018. b. Values calculated as the difference of the 2018 projection and values at the time of DPF approval (Sept. 2012). 1.6 Against this backdrop, a new government led by President Otto Perez Molina, who took office in January 2012, declared its intention to implement tax reform to broaden the tax base, eliminate exemptions, and improve revenue collection. However, results were disappointing: in 2013, both revenue collection and budgetary cash flow were below expectations. By 2014, government revenues had increased by only 0.25 percent of GDP rather than the 1.25 percent that was planned (table 1.2; IMF 2014, 7). Despite maintenance of low fiscal deficits, the government experienced a cash flow crisis, which provided additional motivation to seek budget support from the World Bank. 1.7 The World Bank responded by preparing a programmatic series of two budget support operations: Fiscal Space for Greater Opportunities (DPL 1, $200 million), approved on September 27, 2012, and Enhanced Fiscal and Financial Management for Greater Opportunities (DPL 2, $340 million), approved on June 17, 2014. The DPLs were made effective and disbursed on approval. No International Monetary Fund (IMF) program was in place during this period, although the World Bank consulted the IMF during preparation and during the policy dialogue. 4 2. Relevance of the Objectives and Design Objectives 2.1 The series’ development objectives were to “(i) strengthen tax administration and tax policy; (ii) strengthen budget management and increase the results orientation of public spending; and (iii) improve the management and coordination of social policies” (World Bank 2014, iv, 9). They were substantially relevant at the time of preparation and remain so today. They were consistent with the government’s agenda to promote growth and social development and with the pillars of the Country Partnership Strategy, presented to the World Bank Board of Executive Directors (the Board) jointly with DPL 1. The relevance of the objectives is further highlighted by the fact that for decades Guatemala has had one of the lowest revenue-to-GDP ratios in the world and, therefore, very limited resources with which to address pervasive poverty and extreme social inequities. The objectives of the series were also linked to the government’s Fiscal Pact and Zero Hunger Pact targeting improvements in fiscal and nutrition outcomes. However, the objectives would have been even more relevant if they had been less broad and more precisely formulated.3 For example, the first objective is so broad that almost any fiscal measure could be conceptually related to it, yet it would be difficult to argue that any single measure could clearly contribute in a meaningful way to such a broad objective. Increasing government revenues, by contrast, is a clear objective that would have made the associated results framework and measurement of its achievement more logical and transparent. Design 2.2 Relevance of design is rated modest. The theory of change was unclear, with weak links between objectives, activities, and indicators. Design appears to have been based on the assumption that policy reform is largely a straightforward undertaking, in which measures result in outcomes without the intrusion of governance issues. It was assumed that laws that were passed would be implemented. Thus, the strengthened tax policy and tax administration objectives were to be implemented through some legislative reforms and executive decisions at the level of the Ministry of Finance to reduce some exemptions, simplify rates, and increase registration of taxpayers.4 This assumption was a significant flaw because, in Guatemala, governance issues underlying policies are critical factors that affect policy implementation, especially in the realm of taxation. 2.3 Prior actions were mostly process, output, and upstream activities, with some of them easily reversible and/or requiring considerable follow-up to ensure 5 implementation and desired outcomes. They were also weakly linked to the intended results and associated outcome indicators. For example, tax-related prior actions were linked to legislative reforms and implementation of executive decrees on simplifying rates and eliminating exemptions and specific taxes, but legislation was immediately challenged in the constitutional court, delaying and effectively stymying implementation. Executive decrees were upstream measures, far removed from implementation, given the long history of past reform attempts being blocked by vested interests. Outcome indicators were the income tax–to-GDP ratio and the number of taxpayers, neither of which, in isolation, is an adequate measure of a strengthened tax policy and administration (appendix B). There was no indicator to measure improvements in tax administration. Actions to achieve results-oriented budgeting were introductory steps requiring considerable further measures before any tangible results could be expected. Prior actions to enhance management and coordination of social policies were stronger, resulting in the creation of the Ministry of Social Development with a clear institutional mandate in this area. 2.4 It is unclear whether development policy financing was the most appropriate instrument for supporting such process-oriented, upstream actions, which might have been better supported through technical assistance or capacity-building projects, especially given the small fiscal deficit and the lack of commitment to expand social sector spending. Development policy financing should support policy and institutional reforms with potential to deliver tangible improvements in reform outcomes. Moreover, in Guatemala’s case, the emphasis on process seems especially inappropriate given the country’s poor track record in implementation of taxation-related policy reform. Major governance issues and opposition to reform implementation were well known in the design stage, but there is no evidence that the World Bank planned for or took mitigating measures directly targeting these constraints. Even taking the choice of DPL at face value, there was no targeted technical assistance that would aim to alleviate capacity constraint and/or deal with underlying governance issues in the key reform areas. 2.5 The macroeconomic framework was fiscally sound at the time of approval. The government’s long-standing position on fiscal and monetary policy aimed at low inflation, small fiscal deficits, and limited borrowing. The framework remained adequate throughout the series and subsequently. The budget deficit averaged 1.7 percent of GDP during the 2012–18 period and annual inflation was 3.9 percent. The public debt–to-GDP ratio remained unchanged at about 24.5 percent. However, very low government revenues, at 11.6 percent of GDP in 2012––declining to 10.6 percent in 2018—constrained capital expenditures, which remained at less than 2.5 percent of GDP during most of the evaluation period. Social expenditures were inadequate, especially 6 given the scale of poverty, exclusion, and inequities in Guatemala. There was no IMF program. In 2012, in parallel to the World Bank program, the Inter-American Development Bank (IDB) provided hybrid financing, combining budget support and investments in support of fiscal consolidation.5 3. Achievement of the Objectives 3.1 The achievement of program development objectives was modest. This rating is based on updated results indicators using data from mission interviews and subsequent IMF and World Bank reports (table 3.1). It reflects modest achievement in strengthening tax administration, tax policy, and results budgeting, and substantial achievement in coordination of social policy. Strengthening Tax Administration and Tax Policy 3.2 The achievement on tax administration and tax policy was modest. The first outcome indicator was an increase in the income tax–to-GDP ratio from a baseline of 2.7 percent in 2011 to 3.2 percent by the end of 2014; it was not achieved according to the latest data available at the end of 2018 (table 3.1). Although the ratio improved in 2014, this was not sustained, and by 2018 it had fallen to 2.8 percent, barely above the baseline. Indirect tax revenues and overall tax revenues as a percentage of GDP also declined— the total revenue–to-GDP ratio in 2018 stood at 10.6 percent, a full percentage point lower than at the time of the approval of the series in 2012.6 Other measures of indirect tax revenues and overall revenues, which would have been more appropriate measures of broad tax effort, also declined (table 1.1). The second indicator was an increase in the tax base (measured as the number of effective taxpayers making direct payments to the Superintendencia de Administración Tributaria [tax administration service; SAT]) by at least 10 percent from 2011 to 2014 (baseline: 2011 = 1.44 million; target: 2014 = 1.58 million). This was achieved and sustained, with 2.2 million taxpayers registered in 2018. The third indicator was the increase in the value of administrative sanctions in the area of customs in line with the National Customs Law (baseline: 2013 = 0; target: 2014 = Q 4 million). Although the target was achieved in 2014, the customs service has subsequently been unable to carry out its functions due to a series of corruption scandals. Therefore, no data are available on this indicator as of 2018, and its continued achievement after 2014 could not be verified. The fourth indicator was the increase in the number of countries with which Guatemala has a signed agreement to exchange tax- related information (baseline: 2011 = 0; target: 2014 = 60); and the number of countries with which Guatemala is exchanging tax-related information, on request, through information-sharing agreements or through the Organisation for Economic Co-operation and Development Multilateral Convention (baseline: 2011 = 0; target: 2014 = 5). Again, 7 no updated information was available. Moreover, it is not clear how relevant this indicator is to achieving the objective of strengthening Guatemala’s tax policy and tax administration with a view to increasing revenues. As noted earlier, there is no indicator to measure improvements in tax administration. 3.3 The main reason for lack of achievement was the opposition to tax reform reflected in strong private sector opposition, delays in parliamentary approval of the tax reform legislation, and the failure to implement many measures that had been legislated or decreed. The World Bank expected this strong opposition but underestimated the opposition’s ability effectively to block tax reform, despite similar impediments to previous attempts. The World Bank showed excessive optimism about the new government’s chances of overcoming opposition from vested interests. 3.4 Although the counterfactual is unknown, there are some indications from mission interviews that revenue performance might have been even worse in the absence of the 2012–13 tax reforms, but this is, of course, conjectural. 3.5 On the positive side, the World Bank had an innovative technical dialogue with the SAT, using behavioral insights to help improve marginal tax compliance of delinquent taxpayers (see Box 3.1). Although these efforts may have yielded some results in the short term, their overall impact on revenues over the longer term has not been large, since they targeted only identified delinquent taxpayers and past taxes due on the margin. Nevertheless, the SAT has taken up the approach and has continued using it in subsequent years. Moreover, the intervention has been emulated in some other countries (see Box 3.1). Box 3.1. What Works: Using Behavioral Insights and “Nudges” to Improve Tax Compliance The World Bank worked with the Superintendencia de Administración Tributaria (tax administration service; SAT) in using behavioral insights and experience from the United Kingdom to help improve tax compliance in Guatemala. The objective was to adopt simple and almost costless interventions that had the potential to improve compliance through modifying standard SAT letters to taxpayers and specifically through invoking social norms and potential audits. In 2015, SAT rolled out a nationwide experimental campaign designed to measure the impact of specific messages sent in letters to delinquent income tax payers. The letters attempted to convey what was thought to be the most effective message in the Guatemalan context. The intervention “worked” in the sense that collection from delinquent taxpayers increased. It was found that letters invoking social norms and audits were four times more likely to result in increased compliance than other letters. The SAT has since adopted, modified, and used such experiments on its own. After the Guatemala experience, similar interventions were made in other countries, such as Albania and Costa Rica. Sources: Calvo-González, Cruz, and Hernandez 2018; Calvo-González and Zoratto 2017. 8 Table 3.1. Guatemala DPL Series: Achievement of Target Indicators under the First Two Objectives Indicator Baseline Original Target Actual Value Updated Value Indicator Increase in the income tax-to-GDP ratio 1: Value 2.7 % GDP 3.2% GDP 3.1% GDP 2.8% GDP Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Source: http://www.minfin.gob.gt/images/archivos/estadisticas/doc108.pdf [17-apr-2019] Indicator Increase in the tax base (number of effective tax payers making direct payments to SAT) by at least 10 2: percent from 2011 to 2014 Value 1,441,246 1,585,370 1,658,765 2,228,549 Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Source: chrome-extension://bpmcpldpdmajfigpchkicefoigmkfalc/views/app.html [17-apr-2019] Indicator Increase in the amount of administrative sanctions in the area of customs in line with the National 3: Customs Law Value 0 Q. 4 mill Q. 7.2 mill n/a Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Source: Indicator Increase in the number of countries with which Guatemala has a signed framework to exchange tax 4: related information; (b) increase in the number of countries with which Guatemala is exchanging tax related information, upon request, through information sharing agreements Value 0 60 & 5 94 & 13 n/a Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Commen OCDE Press release indicates that convention includes 111 jurisdictions ts http://www.oecd.org/countries/guatemala/guatemala-strengthens-international-tax-co-operation- Source: ratifies-the-convention-on-mutual-administrative-assistance-in-tax-matters.htm Indicator Increase in the percentage of children under 1 year old in 83 prioritized municipalities who receive the 5: appropriate growth promotion package of services for their age which include weight and height check-ups Value 37.50% 50% 87% n/a Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Commen Indicators found are relative to vaccination rates and height check-up for five years old. ts Source: Indicator Increase in the percentage of the total budget under the results-based budgeting framework 6: Value 0 9% 9.50% 39.27% Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Commen IEG team's calculations based in the information available on Minfin´s web page. ts Source: www.mingin.gob.gt Sources: Independent Evaluation Group; government of Guatemala data. Note: IEG = Independent Evaluation Group; Minfin = Ministry of Public Finance; n/a = not available; OECD = Organisation for Economic Co-operation and Development; SAT = Superintendencia de Administración Tributaria (tax administration service). 9 Strengthening Budget Management and Increasing Results Orientation of Public Spending 3.6 Achievement under this objective was modest. The DPL series aimed to promote greater accountability and outcome orientation of public spending. Previous experience with a World Bank–financed emergency operation (after the eruption of a volcano in 2010) revealed weak monitoring and accountability mechanisms, reflected in difficulties in accounting for public expenditures allocated to beneficiaries of the emergency operation.7 The introduction of results-based budgeting started with the health sector, with the intention of applying its experience to other line ministries later. 3.7 The first outcome indicator (indicator 5, Table 3.1) was an increase in the percentage of children under one year old in 83 prioritized municipalities who receive the appropriate growth promotion package of services for their age, which includes weight and height check-ups (baseline: 2011 = 37.5 percent; target: 2014 = 50 percent). Although the target appeared to have been exceeded in 2014, no further data on this indicator are available. Data exist for children under the age of five, but they are only marginally related to the effectiveness of basic health and growth promotion services for infants. Stunting rates for children under five years are particularly high among the poor (66 percent), rural dwellers (59 percent), and indigenous groups (61 percent), reflecting widespread malnutrition among infants and in early childhood.8 3.8 The second indicator (indicator 6, table 3.1) was an increase in the percentage of the total budget prepared using the results-based budgeting framework (baseline: 2011 = 0 percent; target: 2014 = 9 percent). As of 2018, this target appeared to have been surpassed, with 39 percent of total budget expenditures covered under the formal results-based budgeting framework. However, mission discussions indicate that, although progress may have been made in the formal adoption of results-oriented budgeting, practices and outcomes do not reflect this, especially outside the Ministries of Finance and Health. The 2018 Public Expenditure and Financial Accountability assessment shows the lowest scores on indicators related to results budgeting and medium-term fiscal planning, policy, and budgeting (Giussani, Guardiola, and Ospina 2018). Form does not replace substance, and the attainment of the indicator target should be viewed in that light. 10 Table 3.2. Guatemala DPL Series: Achievement of Target Indicators under the Third Objective Indicator 7: Increase in the percentage of beneficiaries across all social programs that are included in the Unique Beneficiary Registry Value 0 80% 90% 100% Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 Press release indicates the existence of an information system "Registro Único de Usuarios Nacional -RUUN-" that consolidates the Comment information of beneficiaries across all the social programs. Source: http://www.mides.gob.gt/webtwo/mides-capacita-personal-para-analizar-programas-sociale Indicator 8: Increase in the percentage of the population in the country represented by an active COMUSAN that is in charge of coordinating the implementation of the Zero Hunger Pact at the local level Value 25% 90% 100% 99.41% Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-18 In 2018 SESAN reported the existence of 338 comusan, currently the Comment country is divided in 400 municipalities. Source: http://www.sesan.gob.gt/wordpress/wp-content/uploads/2019/02/Memoria-de-Labores-20 Indicator 9: CES holds regular meetings to discuss public policies, and has issued consensus resolutions on policy issues that have been supported by its members Value No Yes Yes Yes Date of reference 31-Dec-11 31-Dec-14 31-Dec-14 31-Dec-17 Source: https://ces.gob.gt/wp-content/uploads/2019/01/MEMORIA-DE-LABORES.pdf [18-april-2019] Note: CES = Consejo Económico Social (Economic and Social Council); COMUSAN = Comisión Municipal de Seguridad Alimentaria y Nutricional del Municipio (Municipal Commission on Food Security and Nutrition); SESAN = Secretaría de Seguridad Alimentaria y Nutricional. Improving the Management and Coordination of Social Policies 3.9 Achievement under this objective was substantial. The reforms supported included the creation of the Ministry of Social Development, the development of a national social information system, and the establishment of the Consejo Económico Social (Economic and Social Council; CES) as a consultative body bringing together representatives of the private sector, government, and civil society and indigenous groups to discuss policy and reform issues, which was lacking in Guatemala’s deeply fragmented and divided polity. The CES was also seen as a way to raise the profile of pressing social issues that did not get sufficient attention in the priority policy agenda. 3.10 The first outcome indicator (indicator 7, Table 3.2) was an increase in the percentage of beneficiaries across all social programs who are included in the Unique Beneficiary Registry (baseline: 2011 = 0; target: 2014 = 80 percent). The target has been achieved as there now exists a unique registry that includes beneficiaries from all social 11 programs. This action was a necessary condition for efficient administration and targeting. The second indicator (indicator 8, Table 3.2) was an increase in the percentage of the population in the country represented by an active community organization (Municipal Commission on Food Security and Nutrition) that is in charge of coordinating the implementation of the Zero Hunger Pact at the local level (baseline: 2011 = 25 percent; target: 2014 = 90 percent);9 the indicator was also achieved. The third indicator (indicator 9, Table 3.2), that the CES hold regular meetings to discuss public policies and issue consensus resolutions that have been supported by its members, was also achieved, although the field interviews raise questions about the frequency and effectiveness of these meetings as a mechanism for coordination. Mission discussions also indicate that there has not been a noticeable improvement in the coordination of views expressed by the nongovernment stakeholders, who anticipated that a stronger voice for the vulnerable groups in the CES would result in more tangible results—for example, improved nutrition and health outcomes. Instead, these continue to be poor. Another weakness is that there are no data available or any indicators to measure improved management of social programs, one of the two dimensions of the objective. 3.11 Despite the shortcomings noted, however, the actions and achievements in the three results areas are overall deemed substantial, as they go in the desired direction— toward strengthening of social policies for effective poverty reduction—in the early stage of long-term reform. 4. Outcome 4.1 The outcome of the DPL series is moderately unsatisfactory, reflecting substantial relevance of objectives, modest relevance of design, and modest achievement of results. Program objectives were weakly related to actions, and results indicators had significant weaknesses in that they were mostly upstream and process oriented; other indicators not included could have measured important elements of the objectives (for example, tax administration and management of social programs). The impact of reforms appears limited. Revenue collection did not improve. Strong opposition to tax reform from vested interests, including constitutional challenges to tax reform and parliamentary opposition, delayed and undermined implementation. Although results- based budgeting was formally adopted across several ministries, there are no indications that this has resulted in greater accountability and efficiency in the use of resources. Coordination-related outcome targets under the third objective were achieved, but there is, thus far, no evidence of improved results in the field, nor of enhanced management of social programs. 12 Risk to Development Outcome 4.2 The risk to development outcome is high. Government revenues in relation to GDP decreased rather than increased as the program intended. This major issue therefore continues to affect adversely all other economic and social reforms, and in particular, two key dimensions of the series. Lack of government funding constrains the implementation of social policies, as well as investments in better data, monitoring and evaluation (M&E) systems, skills, and capacity necessary for effective implementation of results-based budgeting. As noted in the Implementation Completion and Results Report Review, the issue also is a source of macroeconomic vulnerability; the government budget remains highly sensitive to natural disasters and external shocks, with potentially significant economic and social impacts. Without budgetary buffers and insurance mechanisms, the ability of the government, private sector, and households to cope with such shocks is severely limited. Bank Performance Quality at Entry 4.3 Bank performance was moderately unsatisfactory. On the positive side, there was considerable analytical work that informed the DPL series, including on government revenues, results-based budgeting, and social programs (World Bank 2012b). There was good coordination with the IDB, which had a parallel operation supporting fiscal consolidation, and with the IMF, although there was no IMF program in place. 4.4 However, the World Bank’s approach to the design of the key tax reforms was overly technical. Insufficient account was taken of political economy considerations and of the strength of the opposition from vested interests, which was able to stall the reforms through constitutional challenges and parliamentary obstruction. Governance issues were, however, long-standing and well known to the World Bank at the time of the preparation of the series. There was awareness that reforms would probably be challenged in the courts and in Parliament. In fact, as part of its internal review, the World Bank sought a legal expert opinion on the probability that revenue reforms would be approved by Parliament before the first loan was considered by the Board, and the opinion was positive. With hindsight, the World Bank perhaps could have aimed to alleviate political economy constraints by convening the main stakeholders to seek common ground and help build consensus for reform. Alternatively, it could have provided more parallel technical assistance to alleviate capacity constraint. 4.5 There was no IMF program. The IMF did not provide a comfort letter to the World Bank for the series (although both Article IV Consultations and IMF Executive 13 Board discussions persistently emphasized the criticality of strengthening revenue mobilization for lasting fiscal and social sustainability). The decision nevertheless to proceed reflected the perception that this was a high-risk but potentially high-reward series. Moreover, the tax reforms adopted by the new government in 2012 were wider in scope than prior efforts. Although the government did face cash flow problems, there was a sense that there was no threat of sovereign default, so this possibility did not seem to be a factor in the World Bank’s decision to lend. It is noteworthy that the IDB’s parallel operation faced similar opposition. 4.6 The M&E design had significant shortcomings, such as the absence of any indicator measuring improvement in tax administration (see the “Monitoring and Evaluation: Design” section). 4.7 As noted, although the macroeconomic framework was formally adequate for the purposes of development policy lending, there was a serious and long-standing issue regarding how macroeconomic stability was maintained––via very low government revenues and low and inadequate social and public investment spending and borrowing. This issue, in turn, affected achievement of results-based budgeting and the social policy agenda. World Bank Supervision 4.8 Two supervision reports (World Bank 2012d and 2013b) and the program document for DPL 2 (World Bank 2014) reflect the World Bank’s continued dialogue and attention to the reform agenda. They also report on the delays to the effectiveness of the first operation because of the slow parliamentary approval of the loan. However, supervision missions could not resolve design weaknesses, especially the consequences of insufficient attention to political economy issues, as these were beyond their control. Stronger triggers on tax reform for the second operation, for example, could have tested the readiness for additional budget support. Borrower Performance Government Performance 4.9 Government performance is rated moderately unsatisfactory, mainly on implementation/achievement grounds. On the positive side, the government carried out the prior actions and consulted widely with the World Bank, the IDB, and the European Union. The tax administration service also engaged the World Bank in the innovative intervention on tax compliance that achieved some results on the margin and that seems to have institutional traction. However, implementation of key reforms was lacking; and on the main objective of the series, stronger government revenues, performance was 14 worse in 2018 than in 2012, when the first loan in the series was approved. The lack of support for reforms in the Parliament and among private sector groups remained a key constraint on reform implementation. It appears that there was no systematic effort to engage and convene these different groups to help build consensus for reform. Monitoring and Evaluation Design 4.10 M&E quality is rated modest. Objectives were too broad and weakly related to indicators. Although the overall revenue indicator was useful as a measure of the overall progress in tax policy and tax administration reform, it was subsequently replaced by a narrower measure. There was no indicator for improved tax administration. Results- based budgeting indicators were also process oriented and upstream and could not measure actual implementation of the framework. The indicator on international tax agreement was only weakly related to the tax policy and administration goals. Social policy indicators were useful in gauging progress on coordination, but they did not reflect results in the field. There was no indicator measuring the quality of social program management. Implementation and Use 4.11 The Ministry of Finance oversaw implementation and use of M&E indicators under the DPL series. It has not been possible to document the extent to which M&E data were used for policy making or communication with reform stakeholders, but the government routinely communicated with the World Bank, IDB, and IMF on these and related indicators and reforms. 5. Lessons 5.1 The Independent Evaluation Group draws three lessons from the development policy operation series. They are additional to the lessons provided earlier, in the Implementation Completion and Results Report Review stage, which largely remain valid today.10 • Tax administration and tax policy reforms in the face of major governance issues and long-standing opposition from influential interest groups are unlikely to be successful, even if backed by the World Bank’s analytical support, policy dialogue, and financing; under these conditions, directly and indirectly targeting the governance issues over a longer period would be more appropriate. 15 • Achieving progress on results budgeting requires strengthening of capacity, political commitment, sound M&E indicators, and cross-agency collaboration. • Achieving results in policy lending requires a sound results framework, a credible theory of change, close linking of objectives with policy actions, and outcome-oriented target indicators. 1For more information, see the Worldwide Governance Indicators website at https://info.worldbank.org/governance/wgi/#home. 2The International Monetary Fund states that “higher social and infrastructure spending that would raise government expenditure to at least 15 percent of gross domestic product (GDP) is key to lifting more Guatemalans out of poverty” (IMF 2018b). 3 Insufficiently operational and specific objectives have been a source of weak design in many development policy loans, making it difficult to link them to prior actions and results frameworks in a convincing theory of change (World Bank 2016a). See also World Bank 2016d. 4Although reform of legislation commendably targeted reduction of exemptions, simplification of taxes, and the increase in registration of taxpayers, laws were immediately challenged in the constitutional court and implementation was very limited. 5 The Inter-American Development Bank (IDB) approved an operation aimed at promoting fiscal sustainability. The results indicators of the program measured reduction of the debt level and fiscal deficit. The instrument was hybrid, including a budget support section of $234 million, and an investment section of $3.2 million. The program was approved in August 2012 and the budget support was disbursed in 2013, with the investment section ending its disbursement in 2017. 6To put this in context, the tax revenue–to-GDP ratios during this period for other Central American countries were 13.8 percent for Costa Rica; 17.8 percent for El Salvador; 17.3 percent for Honduras; and 16.5 percent for Nicaragua. 7In 2011, the World Bank approved the $100 million Emergency Support for Social Services project with the objective to preserve health and education services in disaster-stricken areas after severe flooding and other natural disasters that had taken place in Guatemala a year earlier. 8In response to the continued malnutrition crisis among infants, in 2017 the World Bank approved the $100 million Crece Sano nutrition project targeting malnutrition in the first 1,000 days of children’s lives. 9For more information on the Municipal Commission on Food Security and Nutrition, see the organization website at https://comusanguatemala.wordpress.com/quienes-somos. 10The following three lessons were drawn in the Implementation Completion and Results Report Review (World Bank 2016c): (i) weak governance continues to undermine implementation of tax policy reforms; (ii) the data needed to monitor and evaluate an objective to improve budget 16 management (that is, data such as that collected by a Public Expenditure and Financial Management Assessment) are lacking, so that results from the changes in the budgetary framework could not be fully assessed; and (iii) social expenditures could be made more effective by integration across social programs and across government levels. 17 Bibliography Barrientos, R. 2012. Análisis breve de la reforma tributaria guatemalteca 2012. Guatemala City: Instituto Centroamericano de Estudios Fiscales. https://www.icefi.org/sites/default/files/analisis_reforma_tributaria_en_guatemala_2012. pdf. Calvo-Gonzáles, Oscar, Abel Cruz, and Marco Hernandez. 2018. “The Ongoing Impact of ‘Nudging’ People to Pay Their Taxes.” Voices (blog), December 4. http://blogs.worldbank.org/voices/ongoing-impact-nudging-people-pay-their-taxes. Calvo-González, Oscar, and Laura Zoratto. 2017. Behavioral Insights for Development: Cases from Central America. Directions in Development. Washington, DC: World Bank. Department of Public Credit. 2019. “Reporte de operaciones de crédito público.” Ministry of Public Finance, Guatemala City. http://dcp- web.minfin.gob.gt/Documentos/Estadisticas/Reporte_Operaciones_Credito_Publico.pdf. Economic and Social Council. 2012. “Legislación CES.” https://ces.gob.gt/ley -del-ces. ———. 2014. Informe 2013–2014: El diálogo Social y las Póliticas públicas en Guatemala. Guatemala City, Guatemala: Consejo Económico Social. https://ces.gob.gt/wp- content/uploads/2019/01/final.pdf. ———. 2017. Plan de reactivación económica. Guatemala City, Guatemala: Consejo Económico Social. https://ces.gob.gt/wp-content/uploads/2019/02/Documento-Reactivacio%CC%81n- Econo%CC%81mica-CES-2017.pdf. ———. 2018. Memoria de labores 2017. Guatemala City, Guatemala: Consejo Económico Social. https://ces.gob.gt/wp-content/uploads/2019/01/MEMORIA-DE-LABORES.pdf. ———. 2019. “Opiniones a legislación del CES.” https://ces.gob.gt/opiniones. Giussani, B., U. Guardiola, and J. Ospina. 2018. III Informe de desempeño de la gestión de las finanzas públicas (PEFA). Washington, DC: Public Expenditure and Financial Accountability. Guatemala, Ministry of Public Finance. 2016. “Aprendiendo sobre el proceso presupuestario.” Ministry of Public Finance, Guatemala City. http://www.minfin.gob.gt/images/downloads/proceso_presupuestario/aprendiendo_141 216.pdf. ———. 2019a. “Ingresos financieros como % del PIB.” Ministry of Public Finance, Guatemala City. http://www.minfin.gob.gt/images/archivos/estadisticas/doc108.pdf. ———. 2019b. “Presupuesto general de ingresos y egresos del estado para el ejercicio fiscal 2019.” Ministry of Public Finance, Guatemala City. http://www.minfin.gob.gt/images/archivos/presua2019/inicio%201.htm. 18 ———. 2019c. “Sistema de contabilidad Integrada.” Ministry of Public Finance, Guatemala City. https://sicoin.minfin.gob.gt/sicoinweb/login/frmlogin.htm. ———. 2019d. “Situación financiera como % del PIB.” Ministry of Public Finance, Guatemala City. http://www.minfin.gob.gt/images/archivos/estadisticas/doc18.pdf. Guatemala, Ministry of Social Development. 2018. “Mides capacita personal para analizar programas sociales.” Ministry of Social Development, Guatemala City. http://www.mides.gob.gt/webtwo/mides-capacita-personal-para-analizar-programas- sociales. Guatemala, Secretaría de Seguridad Alimentaria y Nutricional. 2018. Memoria de labores 2018. http://www.sesan.gob.gt/wordpress/wp-content/uploads/2019/02/Memoria-de-Labores- 2018.pdf. Instituto Centroamericano de Estudios Fiscales. 2016. “Hoja de ruta y propuesta de reformas a la ley orgánica de la SAT.” Noticio February 19, Instituto Centroamericano de Estudios Fiscales. https://www.icefi.org/noticias/icefi-presenta-version-final-de-hoja-de-ruta-y- propuesta-de-reformas-ley-organica-de-la-sat. IDB (Inter-American Development Bank). 2016a. Comparative Evaluation: Review of Bank Support to Tax Policy and Administration, 2007–2016. Washington, DC: IDB. doi:http://dx.doi.org/10.18235/0000833. ———. 2016b. Country Program Evaluation: Guatemala 2012–2016. Washington, DC: IDB. https://publications.iadb.org/en/country-program-evaluation-guatemala-2012-2016. ———. 2018a. “Alcance logrado en resultados y productos.” Excel document, Inter-American Development Bank, Washington, DC. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=EZSHARE-1210624942-45. ———. 2018b. “Factores críticos que afectaron el desempeño del programa .” Excel document, Inter-American Development Bank, Washington, DC. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=EZSHARE-1210624942-46. ———. 2018c. “PCR Programa de Consolidación fiscal para Guatemala GU -L1064 y GU-L1160, IADB.” Excel document, Inter-American Development Bank, Washington, DC. http://idbdocs.iadb.org/wsdocs/getdocument.aspx?docnum=EZSHARE-1305285792-4. IMF (International Monetary Fund). 2013. “Guatemala: 2013 Article IV Consultation.” IMF Country Report 13/247, IMF, Washington, DC. ———. 2014. “Guatemala: 2014 Article IV Consultation—Staff Report; Press Release; and Statement by the Executive Director for Guatemala.” IMF Country Report 14/287, IMF, Washington, DC. 19 ———. 2016. “Guatemala: 2016 Article IV Consultation—Press Release; Staff Report; Informational Annex.” IMF Country Report 16/281, IMF, Washington, DC. ———. 2018a. “Guatemala: 2018 Article IV Consultation-Press Release; Staff Report; and Statement by the Executive Director for Guatemala.” IMF Country Report 18/154, IMF, Washington, DC. ———. 2018b. “Guatemala: More Investment and Social Spending Needed. ” IMF Country Focus. https://www.imf.org/en/News/Articles/2018/05/30/NA060118-Guatemala-More- Investment-and-Social-Spending-Needed. ———. 2019. “Growth Slowdown, Precarious Recovery.” World Economic Outlook. https://www.imf.org/en/Publications/WEO/Issues/2019/03/28/world-economic-outlook- april-2019. International Food Policy Research Institute. 2016. Informe final de evaluación de impacto del Plan del Pacto Hambre Cero. Washington, DC: International Food Policy Research Institute. http://www.siinsan.gob.gt/siinsan/wp-content/uploads/Informe-Final-Evaluacion- Impacto-PPH0.pdf. Organisation for Economic Co-operation and Development. 2017. “Guatemala Strengthens International Tax Co-operation—Ratifies the Convention on Mutual Administrative Assistance in Tax Matters.” OECD, Paris. http://www.oecd.org/countries/guatemala/guatemala-strengthens-international-tax-co- operation-ratifies-the-convention-on-mutual-administrative-assistance-in-tax- matters.htm. ———. 2019. “Convention on Mutual Administrative Assistance in Tax Matters.” OECD, Paris. http://www.oecd.org/tax/exchange-of-tax-information/convention-on-mutual- administrative-assistance-in-tax-matters.htm. Organización Internacional del Trabajo. 2012. “Historia cooperativa en Guatemala.” https://www.aciamericas.coop/IMG/pdf/wcms_185442.pdf. Sanchez, Susana M., Kinnon Scott, and J. Humberto Lopez. 2016. Guatemala: Closing Gaps to Generate More Inclusive Growth. Systematic Country Diagnostic. Washington, DC: World Bank. License: Creative Commons Attribution CC BY 3.0 IGO. SAT (Superintendencia de Administración Tributaria). 2019. “Contribuyentes efectivos .” Excel document, SAT, Guatemala City. https://portal.sat.gob.gt/portal/descarga/1741/estadisticas-tributarias/5203/cefec.xlsx. World Bank. 2005. Guatemala Country Economic Memorandum: Challenges to Higher Economic Growth. Washington, DC: World Bank. ———. 2012a. “Guatemala—Review of Country Partnership Strategy for FY09–12.” Country Partnership Strategy Completion Report Review 72850, Independent Evaluation Group, World Bank, Washington, DC. 20 ———. 2012b. “Guatemala—First Programmatic Development Policy Loan.” Program Document Report 70208-GT, World Bank, Washington, DC. ———. 2012c. Guatemala—Public Expenditure Review. Washington, DC: World Bank. ———. 2012d. Implementation Status & Results, Guatemala–– First Programmatic Development Policy Loan. (December 19). Washington, DC: World Bank. ———. 2013. Guatemala—Country Partnership Strategy, FY13–16. Washington, DC: World Bank. ———. 2013b. Implementation Status & Results, Guatemala–– First Programmatic Development Policy Loan. (December 19). (July 11). Washington, DC: World Bank. ———. 2014. “Guatemala—Enhanced Fiscal Space for Greater Opportunities Development Policy Loan.” Program Document Report 85266-GT, World Bank, Washington, DC. ———. 2016a. “Guatemala—Review of the Country Partnership Strategy for FY13–16.” Completion and Learning Report Review 110071, Independent Evaluation Group, World Bank, Washington, DC. ———. 2016b. “Guatemala—Fiscal Space for Greater Opportunities.” Implementation Completion and Results Report ICR3756, World Bank, Washington, DC. ———. 2016c. “Guatemala—Fiscal Space for Greater Opportunities.” Independent Evaluation Group, Implementation Completion and Results Report Review ICRR0020237, World Bank, Washington, DC. ———. 2016d. “Strengthening Results Frameworks in Development Policy Operations.” IEG blog, May 31. Independent Evaluation Group, World Bank. https://ieg.worldbankgroup.org/blog/strengthening-results-frameworks-development- policy-operations. ———. 2017. “Guatemala: Crece Sano—Nutrition and Health Project.” Project Appraisal Document 1922, World Bank, Washington, DC. 21 Appendix A. Basic Data Sheet Fiscal Space for Greater Opportunity First Programmatic Development Policy Loan (IBRD-82030; P131763) Table A.1. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percentage Financing ($, millions) ($, millions) of Appraisal Estimate Total project costs 200 200 100 Loan amount 200 200 100 Cofinancing -- -- -- Cancellation 0 0 0 Note: -- = not available. Table A.2. Cumulative Estimated and Actual Disbursements Disbursements FY14 Appraisal estimate ($, millions) 200 Actual ($, millions) 199.50 Actual as percentage of appraisal 100 Date of final disbursement 12/2013 Note: FY = fiscal year. Table A.3. Project Dates Event Original Actual Concept review 06/28/2012 06/28/2012 Board approval 09/27/2012 09/27/2012 Signing 11/13/2013 11/13/2013 Effectiveness 11/21/2013 11/21/2013 Closing date 12/31/2013 12/31/2013 23 Table A.4. Staff Time and Cost World Bank Budget Only Staff time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY13 22.46 70.32 Total 22.46 70.32 Supervision or ICR FY13 50.50 175.61 Total 50.50 175.61 Note: FY = fiscal year; ICR = Implementation Completion and Results Report. a. Includes travel and consultant costs. Table A.5. Task Team Members Name Titlea Unit Jasmin Chakeri Senior Economist GMF04 Oscar Calvo-González Practice Manager GPV04 Mateo Clavijo Financial Analyst GFM04 Patricia Chacon Holt Language Program Assistant GMF04 Alma Hernandez Executive Assistant CSRVP Daniel Alvarez Senior Public Sector Management GGO16 Specialist Enrique Fanta Senior Trade Facilitation GTC04 Specialist Maria Gonzalez de Asis Lead Operations Officer GGHVP Tracey Hsu Junior Professional Associate LCSPE Leonardo Lucchetti Economist GPV04 Kinnon Scott Senior Economist GPV04 Mauricio Garita LCCGT Katherine Grau LCCGT Fernando Paredes Operations Officer LCCGT Christine Lao Pena LCSHH Edmundo Murrugarra Senior Social Protection GSP04 Economist Jimena Garrote Senior Counsel LEGLE Escarlata Baza LEGLE Rodrigo Serrano-Berthet LCSSO Concepcion Aisa Senior Financial Officer FABBK Rodrigo Cabral Senior Financial Officer FABDM 24 Name Titlea Unit Raul Junquera Lead Public Sector Specialist GGO18 Kai Kaiser Senior Economist GGO14 Michele Gragnolati Lead Specialist GHNDR C. Felipe Jaramillo Director, Strategy and Operations MDI Rodrigo A. Chaves Country Director EACIF Auguste Tano Kouame Practice Manager GMF05 Oscar Avalle Manager SECPO Note: a. Co-task team leader Enhanced Fiscal and Financial Management for Greater Opportunities: Second Programmatic DPL (IBRD-83850; P133738) Table A.6. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percentage Financing ($, millions) ($, millions) of Appraisal Estimate Total project costs 340 340 100 Loan amount 340 340 100 Cofinancing -- -- -- Cancellation 0 0 0 Note: -- = not available. Table A.7. Cumulative Estimated and Actual Disbursements Disbursements FY15 Appraisal estimate ($, millions) 340 Actual ($, millions) 339.15 Actual as percentage of appraisal 100 Date of final disbursement 03/2015 Note: FY = fiscal year. Table A.8. Project Dates Event Original Actual Concept review 03/27/2014 03/27/2014 Board approval 06/17/2014 06/17/2014 Signing Effectiveness 02/26/2015 02/25/2015 Closing date 03/31/2015 03/31/2015 25 Table A.9. Staff Time and Cost World Bank Budget Only Staff time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY14 21.96 50.50 Total 21.96 50.50 Supervision or ICR FY14 12.71 53.27 Total 12.71 53.27 Note: FY = fiscal year; ICR = Implementation Completion and Results Report. a. Includes travel and consultant costs. Table A.10. Task Team Members Responsibility or Name Titlea Unit Specialty Lending Marco Antonio Hernandez Senior Economist GMF04 Task Team Leader Mateo Clavijo Financial Analyst GFM04 Miguel Angel Saldarriaga Junior Professional Associate GFM04 Patricia Chacon Holt Language Program Assistant GMF04 Diana Lachy E T Temporary GMFD1 Christine Lao Pena LCSHH Edmundo Murrugarra Senior Social Protection Economist GSP04 Wendy de Leon Consultant GEDDR Fernando Paredes Operations Officer LCCGT Carolina Rendon Senior Public Sector Specialist GGO16 Enrique Fanta Senior Trade Facilitation Specialist GTC04 Katherine Grau LCCGT Javier Baez Senior Economist GPV01 Leonardo Lucchetti Economist GPV04 Maria Eugenia Genoni Economist GPV04 Kiyomi Cadena Consultant GPV04 Kathy Lindert Lead Economist GSPDR Maria Pia Cravero Junior Counsel LEGLE Jimena Garrote Senior Counsel LEGLE Patricia de la Fuente Hoyes Senior Financial Management GGO22 Specialist Robert Montgomery Lead Environment Specialist GEN04 26 Responsibility or Name Titlea Unit Specialty Antonio Blasco Senior Financial Management GGO21 Specialist Concepcion Aisa Senior Financial Officer FABBK David Gould Lead Economist ECACE Marcelo Bortman Senior Public Health Specialist GHN04 Munawer Khwaja Tax Administrator Maryanne Sharp Country Operations Adviser LCC2C Humberto Lopez Country Director LCC2C Auguste Tano Kouame Practice Manager GMF05 Oscar Avalle Manager SECPO Oscar Calvo-González Lead Economist – Sector Leader GPV04 Note: a. At time of appraisal and closure, respectively. 27 Appendix B. Development Policy Loan Policy and Results Matrixes Status at Time of Implementation Policy Actions Supported by Policy Actions Supported by Results Indicators for DPL2 (end- Completion and Results Report DPL 1 DPL2 2014) Review (end-2014) Pillar 1: Strengthening Tax Administration and Tax Policy To raise tax revenues, the borrower To raise tax revenues, the borrower 1. Increase in the income tax–to– Partially achieved has has gross domestic product (GDP) ratio The income tax–to-GDP ratio (baseline: 2011 = 2.7 percent; increased but fell short of the target • Widened the tax base subject to • Issued implementing regulations target: 2014 = 3.2 percent) value by 0.1 percentage points of income tax, through the reduction for (i) the income tax reform, as GDP. of the number of tax exemptions evidenced by the Borrower’s 2. Increase in the tax base (number of • Simplified the tax rates for salaried Executive Agreement No. 213- effective taxpayers making direct Achieved workers 2013; (ii) the VAT reform, as payments to the SAT) by at least The number of effective taxpayers • Increased the tax rate on the evidenced by the Borrower’s 10 percent from 2011 to 2014 making direct payments to SAT circulation of land, sea, and air Executive Agreement No. 5-2013; (baseline: 2011 = 1,441,246; target: increased by 15 percent, from vehicles, as evidenced by and (iii) the tax reform for the new 2014 = 1,585,370) 1,441,246 in 2011 to 1,658,765 in Legislative Decree 10-2012; and tax on motor vehicles’ first 2014. The target was exceeded by reformed the value added tax registration, as evidenced by the 80 percent. (VAT) regime for small taxpayers Borrower’s Executive Agreement by expanding the eligibility criteria No. 133-2012; and (indicative for this VAT regime, as evidenced trigger for the second operation) by Legislative Decree 4-2012 • Improved the organizational structure of the Superintendencia de Administración Tributaria (tax administration service, SAT) in line with international practices, as evidenced by Legislative Decree No. 13-2013 To strengthen the control authority of To strengthen the SAT in alignment 3. Increase in the amount of Achieved the SAT, the borrower has with the Central American Uniform administrative sanctions in the area The amount of administrative Customs Code and its regulations, the of customs in line with the National sanctions in customs, in line with the • Introduced the requirement of borrower Customs Law (baseline: 2013 = 0; National Customs Law, reached Q bank-based transactions above a target: 2014 = 7.3 million in 2014, from a baseline of certain threshold amount to be • Approved the National Customs Q 4 million) zero in 2013. The target was exceeded eligible for tax declaration Law through the enactment of by 80 percent. 28 Status at Time of Implementation Policy Actions Supported by Policy Actions Supported by Results Indicators for DPL2 (end- Completion and Results Report DPL 1 DPL2 2014) Review (end-2014) purposes, as evidenced by Legislative Decree No. 14, 2013, Legislative Decree 4-2012 including (i) a definition of • Aligned the borrower’s sanctions customs infringements and related legislation on customs to the sanctions; (ii) the regulation of the Central American Uniform suspension and cancellation of Customs Code, as evidenced by customs licenses; and (iii) the Legislative Decree 10-2012 establishment of procedures against fraud and contraband (implementing regulation was an indicative trigger for the second operation) To increase transparency and To increase transparency and 4. Increase in the number of countries Achieved exchange of information on exchange of information on with which Guatemala has a signed The number of countries with which international taxation, the borrower international taxation, the borrower framework to exchange tax-related Guatemala has a signed framework to has has information (baseline: 2011 = 0; exchange tax-related information target: 2014 = 60); and the number reached 94 jurisdictions (includes • Signed a separate tax information • Signed an additional tax of countries with which Guatemala State Parties to the Convention as well exchange agreement with seven information exchange agreement is exchanging tax-related as jurisdictions) as of December 2015 countries, as evidenced by the Tax with the Commonwealth of information, on request, through from a baseline of zero (2011). Information Exchange Agreements Australia, dated September 26, information-sharing agreements or • Introduced the concept of transfer 2013, (indicative trigger for the through the Organisation for As of August 2015, Guatemala had pricing in the valuation of second operation) Economic Co-operation and signed one bilateral Double Tax transactions between related • Signed the Convention on Mutual Development Multilateral Convention, eight Bilateral Tax parties (partes relacionadas), as Administrative Assistance in Tax Convention, in line with the Global Information Exchange Agreements, evidenced by Legislative Decree Matters on December 5, 2012 Forum’s standards of transparency and a Mutual Assistance Convention 10-2012 • Created an international taxation and exchange of tax information with four other Central American • Signed a memorandum of unit within the borrower’s Ministry (baseline: 2011 = 0; target: 2014 = countries from a baseline of zero understanding with the United of Public Finance to support the 5) (2011). The target was significantly States, dated May 30, 2012, which exchange of information related to exceeded. provides for the exchange of international taxation, as relevant tax information between evidenced by the borrower’s both countries concerning the Executive Agreement No. 26-2014 declared value of imports and exports 29 Status at Time of Implementation Policy Actions Supported by Policy Actions Supported by Results Indicators for DPL2 (end- Completion and Results Report DPL 1 DPL2 2014) Review (end-2014) Pillar 2: Enhancing Budget Management and Increasing the Results Orientation of Public Spending To improve budget management and To improve budget management and 5. Increase in the percentage of Achieved implement a results-based implement a results-based children under one year old in 83 The percentage of children under one methodology of public expenditure, methodology of public expenditures, prioritized municipalities who year old in the 83 prioritized the borrower’s Ministry of Public the borrower has receive the appropriate growth municipalities who receive the growth Finance and Ministry of Public Health promotion package of services for promotion package of services for and Social Assistance have • Adopted a legal framework for their age, which includes weight their age, which includes weight and results-based budgeting, as and height check-ups (baseline: height check-ups, has increased from • Signed a results-based evidenced by Legislative Decree 2011 = 37.5 percent; target: 2014 = 37.5 percent (2011) to 48 percent in management agreement, dated No. 13-2013 50 percent) 2013 and to 87 percent in 2014. The June 28, 2012, whereby the • Complied with the requirements to target was exceeded by 74 percent. borrower commits to, among expand the application of results- other things, allocating budget based budgeting to the borrower’s toward meeting specific targets for Ministry of Economy and the reproductive and nutritional health Ministry of Sports and Culture Achieved 6. Increase in the percentage of the programs for calendar year 2012 • Established operating and The increase in the percentage of the total budget under the results- coordination mechanisms for total budget under the results-based based budgeting framework results-based budgeting in the budgeting framework reached (baseline: 2011 = 0 percent; target: borrower’s Ministry of Public 9.5 percent in 2014 from a baseline of 2014 = 9 percent) Finance and the Ministry of Public 0 percent in 2011. Health and Social Assistance To strengthen budget management and transparency of public expenditures, the borrower has • Mandated the use of the treasury single account to process budget transactions • Mandated that all entities that execute projects with public funds adequately report their activities to the borrower’s Ministry of Public Finance 30 Status at Time of Implementation Policy Actions Supported by Policy Actions Supported by Results Indicators for DPL2 (end- Completion and Results Report DPL 1 DPL2 2014) Review (end-2014) • Introduced regulations to strengthen the monitoring and evaluation of loans and grants to, and trust funds managed by, public entities • Mandated that all public entities use the Sistema Integrado de Administraci6n Financiera (Integrated Financial Management System) to consolidate budgetary and financial information, as evidenced by Legislative Decree No. 13-2013 • Further measures (program document, World Bank 2014, 25) to strengthen results budgeting were indicative trigger for the second operation. Pillar 3: Improving the Coordination and Management of Social Policies To improve the coordination in the To improve the management of social 7. Increase in the percentage of Achieved design and implementation of social policies, the borrower has beneficiaries across all social The percentage of beneficiaries across policies, the borrower has programs who are included in the 72 social programs in 16 institutions • Created the Ministry of Social • Established a social information Unique Beneficiary Registry. included in the Unique Beneficiary Development, as evidenced by system in the borrower’s Ministry (baseline: 2011 = 0; target: 2014 = Registry increased to 90 percent in Legislative Decree 1-2012, dated of Social Development, including 80 percent) 2014 from a baseline of 0 percent in January 24, 2012, and published in information on social programs 2011. the borrower’s Official Gazette on and policies related to February 7, 2012 beneficiaries, geographic coverage • Adopted the Zero Hunger Pact, as and type of program, and a single evidenced by the Consejo beneficiary registry that includes Nacional de Seguridad Alimentaria information on beneficiaries for at y Nutricional Resolution 01-2012, least 75 social programs (indicative dated June 19, 2012 trigger for the second operation) 31 Status at Time of Implementation Policy Actions Supported by Policy Actions Supported by Results Indicators for DPL2 (end- Completion and Results Report DPL 1 DPL2 2014) Review (end-2014) To strengthen the participation of To support the implementation of the 8. Increase in the percentage of the Achieved trade unions, cooperatives, and the Zero Hunger Pact, the borrower has population in the country The percentage of the population in business sector in public economic represented by an active COMUSAN the country represented by an active and social policy making, the • Established 120 Comisión that is in charge of coordinating the COMUSAN that is in charge of borrower has Municipal de Seguridad implementation of the Zero Hunger coordinating the implementation of Alimentaria y Nutricional del Pact at the local level the Zero Hunger Pact reached • Created the Economic and Social Municipio (Municipal Commission (baseline: 2011 = 25 percent; target: 100 percent in 2015 from a baseline of Council (CES), as evidenced by on Food Security and Nutrition; 2014 = 90 percent) 25 percent in 2011. Legislative Decree 2-2012, dated COMUSAN) groups during 2012 January 24, 2012, and published in and 2013 in prioritized 9. CES holds regular meetings to Achieved the borrower’s Official Gazette on municipalities with the highest discuss public policies and has The CES was created in 2012 to February 23, 2012 incidence of chronic malnutrition issued consensus resolutions on promote multisectoral policy policy issues in Guatemala that have consensus. The CES meets once per been supported by its members month and has a permanent commission that meets on a weekly basis, composed of the CES president, the technical secretary, and representatives from the private sector, cooperatives, and labor unions. 32 Appendix C. List of Persons Met Name Title Institution Government Dorval Carias Ministro de Finanzas Ministerio de Finanzas Públicas Rosa María Ortega Directora de Crédito Público Ministerio de Finanzas Públicas Azucena Ramírez (suplente) Consejo de Cooperación Internacional, Dirección de Crédito Público Ministerio de Finanzas Públicas Luis Javier Ortiz Subdirector de Crédito Público Ministerio de Finanzas Públicas Francisco Javier Ortiz Director de DTP, Crédito Ministerio de Finanzas Públicas Público Rodolfo Orozco Director Ejecutivo CONFECOOP Academia Sigfrido Lee Director del Observatorio Universidad Del Valle de Guatemala Económico Sostenible Nongovernmental Organizations Ricardo Barrientos Economista Sénior Instituto Centroamericano de Estudios Fiscales Abelardo Medina Economista Instituto Centroamericano de Estudios Fiscales Mario Garcia Lara Director Ejecutivo Fundación 2020 Jorge Lavarreda Presidente Centro De Investigaciones Económicas Nacionales Development Partners Gerardo Peraza International Monetary Fund Oscar Lora Rocha Banco Interamericano de Desarrollo Beatrice Bussi Jefe de Delegación Unión Europea World Bank Jasmin Chakeri Task Team Leader World Bank Oscar Calvo-Gonzales Former Lead Economist World Bank Marco Hernandez Task Team Leader World Bank Homa Zahra-Fotouhi Country Manager World Bank Carlos Fernando Paredes Senior Country Operations World Bank Solorzano Officer Auguste Kouame Former Practice Manager World Bank 33 Appendix D. Approach, Methods, and Ratings About This Report The Independent Evaluation Group (IEG) assesses the programs and activities of the World Bank to ensure the integrity of the World Bank’s self-evaluation process, to verify that the World Bank’s work is producing the expected results, and to help improve directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20–25 percent of the World Bank’s lending operations through fieldwork. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which executive directors or World Bank management have requested assessments; or those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government and other in-country stakeholders, interview World Bank staff and other donor agency staff both at headquarters and in local offices as appropriate, and apply other evaluative methods as needed. Each PPAR is subject to technical peer review, internal IEG panel review, and management approval. The PPAR is commented on by the responsible World Bank Country Management Unit and is also sent to the borrower for review. IEG incorporates both World Bank and borrower comments as appropriate, and borrower comments are attached to the document sent to the World Bank’s Board of Executive Directors. After the assessment report is sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers a rigor and flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldbankgroup.org). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance refers to relevance of objectives and of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with World Bank country and sectoral assistance 34 strategies and corporate goals. Relevance of design is the extent to which the project’s design is consistent with stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared with alternatives. The efficiency dimension is not applied to development policy operations, which provide general budget support. Possible ratings for outcome: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. Risk to development outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained. Possible ratings for risk to development outcome: high, significant, moderate, negligible to low, and not evaluable. Bank performance: The extent to which services provided by the World Bank ensured quality at entry of the operation and supported effective implementation. The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. Borrower performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation and complied with covenants and agreements toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible ratings for borrower performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. 35