Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) Report Number : ICRR0020798 1. Project Data Country Practice Area(Lead) Ukraine Macroeconomics, Trade and Investment Programmatic DPL Planned Operations: 2 Approved Operations: 2 Operation ID Operation Name P150313 DPL 1 L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-83920 31-Oct-2014 750,000,000.00 Bank Approval Date Closing Date (Actual) 22-May-2014 31-Oct-2014 IBRD/IDA (USD) Co-financing (USD) Original Commitment 750,000,000.00 0.00 Revised Commitment 750,000,000.00 0.00 Actual 750,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Lodewijk Smets Robert Mark Lacey Malathi S. Jayawickrama IEGEC (Unit 1) PHPROJECTDATATBL Operation ID Operation Name P151479 DPL 2 ( P151479 ) Page 1 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-83920,IBRD-85110 30-Jun-2016 500,000,000.00 Bank Approval Date Closing Date (Actual) 25-Aug-2015 30-Jun-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 500,000,000.00 0.00 Revised Commitment 500,000,000.00 0.00 Actual 500,000,000.00 0.00 2. Program Objectives and Policy Areas a. Objectives According to the Program Documents, the objectives of the series were to (1) promote good governance, transparency and accountability in the public sector; (2) strengthen the regulatory framework and reduce the cost of doing business; and (3) reform inefficient and inequitable utility subsidies while protecting the poor (Summary of the Proposed Program, page iv of Program Documents). The same objectives are stated in the Appendix to the first Loan Agreement (Section 1.5, page 12). b. Pillars/Policy Areas There were three pillars: Pillar A: Promote good governance, transparency and accountability in the public sector. To address fundamental government failures, the program supported reforms to increase transparency and efficiency of public procurement, strengthen public investment management, improve accountability of senior public and elected officials through systematic disclosure, and verification of asset declarations and extending external budget audits to public revenues. Pillar B: Strengthen the regulatory framework and reduce costs of doing business. Given Ukraine’s poor investment climate and stunted growth of the private sector, the reform agenda focused on reducing delays in tax reforms, measures to improve the general business environment and addressing distortions in the agricultural land market. Pillar C: Reform inefficient and inequitable utility subsidies while protecting the poor. The reform Page 2 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) program supported several measures to increase the efficiency and equity of gas and heating subsidies. The program supported resolutions to raise tariffs to cost-recovery levels, strengthen the independence of the energy regulator, restructure (unbundle) a state-owned enterprise in the energy sector (Naftogaz), and supported the introduction of a new social protection program while also increasing the targeting of existing schemes. Finally, the program aimed to strengthen the independence of social inspectors by moving this control function to the central government level. While the Program Objectives were not revised, during the preparation of the second operation, a number of policy areas were adjusted due to the protracted conflict in Eastern Ukraine, a deteriorating macroeconomic environment and the election cycle. First, the restructuring of Naftogaz evolved into a more comprehensive reform of the energy sector. Second, the substance of social protection measures was significantly adjusted. Third, the adoption of a decentralization agenda necessitated dropping the policy action of moving the social inspectors to the central government level. For the second operation, the formulation of 8 out of 11 prior actions – prior actions 1, 2, 4, 5, 6, 7, 8 and 11 – was sharpened in light of further preparatory work and the changing policy environment (see Table 4, page 19, Program Document P151479). c. Comments on Program Cost, Financing, and Dates The loan amount of the first operation was US$ 750 million and was fully disbursed on May 28, 2014. The operation closed on October 31, 2014. Originally, the loan amount of the second operation was planned at US$ 250 million. However, taking into account the worsened macroeconomic environment, the loan size of the second operation was doubled to US$ 500 million and fully disbursed on August 28, 2015. The second operation closed on June 30, 2016 (Program Document P151479, page 7). Given the severity of the macroeconomic adjustment (real GDP dropped by 10 percent in 2015) and the escalation of the conflict in Eastern Ukraine, it was decided to move forward the dates on most results indicators by one year. 3. Relevance of Objectives & Design a. Relevance of Objectives The DPO series was relevant with respect to Ukraine’s development priorities, adapted to country conditions and aligned with the World Bank’s Country Partnership Framework. • At project appraisal, the Ukrainian government expressed a clear need to stabilize the economy, improve Page 3 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) transparency in the use of state funds and initiate structural reforms. The DPO series were well aligned with these priorities, as well as with the 2015-2016 Government Program. • The program was adapted to reflect changing country conditions. For instance, the loan size of the second operation was increased, an extraneous prior action was dropped and the dates on most results indicators were moved forward. • The DPO series was also aligned with the World Bank Group’s FY12-16 Country Partnership Strategy for Ukraine, which focused on two pillars: (1) improving public services and public finances, and (2) improving policy effectiveness and economic competitiveness. Rating Substantial b. Relevance of Design Generally, program objectives were logically linked to relevant policy actions. Regarding the first objective, it was reasonable to expect that legislation to reduce exemptions in public procurement, amendments to improve public investment programs and laws to allow for the verification of asset declarations and extend budget audits lead to improved governance, transparency and accountability in the public sector. Regarding the second objective, streamlining VAT refund claims could be expected to reduce the effective cost of doing business. Furthermore, the enactment of legislation to rationalize permits and licenses and measures to improve investor protection could ease the burden on business. Finally, regarding the third objective, resolutions to increase energy tariffs combined with measures to prevent government from interfering with tariff setting could be expected to lead to improved outcomes in this reform area. The support of the popular Housing and Utility Subsidy – through improved targeting and streamlined application – could be expected help to protect the poor from the utility price increases. 　 A number of complementary factors were important to achieve the intended outcomes. For example, while the DPO series supports the enactment of a law to improve public investment management, improved outcomes in this area critically depend on sufficient human capital, resources and aligned incentives to conduct economic analysis and critically appraise public investment projects. Similarly, the verification of asset declarations would lead to improved governance only if effective mechanisms are in place to hold public officials accountable. Design therefore focused on reform areas with strong pre-existing analytical work and ongoing policy dialogue. Program implementation was also to be supported by technical assistance and other projects, as well as by the engagements of other development partners. 　 Given the volatile political and economic environment at the time of program preparation, and the short preparation time available, design was flexible. While the main reform areas were to remain unchanged, the programmatic DPL approach adopted would enable the program to be fine-tuned in reaction to changing political and macroeconomic circumstances. 　 Page 4 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) The Program Document of the first operation mentions that the macroeconomic framework was deemed adequate for the operation to proceed (Program Document P150313, page 3). In February 2014, the government had requested a Standby Arrangement with the IMF. Nevertheless, the Program Document mentions significant macroeconomic and fiduciary risks because of vulnerabilities to external shocks, banking sector instability, geopolitical tensions, insufficient capacity and willingness to implement reforms with the risk of policy reversals (Program Document P150313, pages 3-9 and 27-28). Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Promote good governance, transparency and accountability in the public sector Rationale Increase transparency and efficiency of public procurement. The amendment of the procurement law supported by the first operation reduced the number of exemptions to competitive procurement from 45 to 15. However, in 2015 the government enacted a new public procurement law, superseding the first operation’s amendment to the old law, and introducing some new amendments. Therefore, any changes in public procurement outcomes are unlikely to be highly attributable to the program’s policy actions. Furthermore, Ukraine saw an actual decline in the share of public procurement (measured by value) subject to competitive bidding, from 35 percent to 2013 to 30.2 percent by end-2015. The ICR notes that this is partly due to an increase in security and utility spending (which are not subject to competitive bidding). However, even when excluding utility and security spending, the share of competitive procurement failed to meet the target of 55 percent. According to information provided subsequently by the project team, "legislative changes resulted in an increase in the number of procuring entities. Since these newly added procuring entities had higher thresholds for application of the Public Procurement Law (PPL), we observe an increase in "no procedure", driving down the value-share of competitive procurements." 　 Public Investment Management. The second operation supported an amendment of the budget law to create a modern and transparent public management investment system that ensures that all public investment projects are subjected to economic analysis and scrutinized by the Ministry of Economic Development. Complementary Bank projects and the engagements of other development partners helped Page 5 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) the Authorities in introducing detailed guidelines and methodology for appraising investment projects. A Committee, consisting of key Ministers and members of the Parliamentary Budget Committee was established to select projects. This procedure should lead to higher quality investment projects that are aligned with strategic objectives (Program Document P151479, page 24).The ICR reports that this Committee reviewed and appraised all longlisted projects submitted by the Ministry of Economic Development in 2015 and 2016 (ICR, page 16), as such formally meeting the target (100 percent new eligible public investment projects appraised and selected through the public investment management system). However, no evidence is provided regarding the implementation of economic analysis, the quality of appraisal, and the alignment to strategic objectives. 　 Financial Disclosures. To deter corruption and mismanagement of public funds, the second operation supported legislation to (1) establish external verification of asset declarations, (2) ensure administrative accountability and (3) establish an independent National Anticorruption Prevention Agency (NACP) responsible for verification. 　 While the legislation passed in 2014, the NACP only got staffed in October 2016, with manual verification starting in the calendar year 2017. Therefore, the target of external verification of all financial disclosures by 2016 has not been met. The ICR also notes that the NACP continues to be hampered by malfunctioning IT and data sharing systems. The project team subsequently observed that "while NACP has had limited impact on administrative accountability (though it has issued a number of administrative notices regarding late declarations and missing or misstated information), the asset declaration system has provided valuable evidence for the efforts of the anti-corruption investigating and prosecuting services] to hold high level officials criminally liable." 　 External Budget Audit. The second operation supported a law – published in the Official Gazette on April 25, 2015 – making the audit of budget revenues an explicit mandate of the Accounting Chamber. The Chamber’s mandate had previously been limited to expenditures 　 Transfer Pricing Law. The DPO supported the enactment of amendments to the Law on Transfer Pricing in order to reduce tax avoidance (IRC, page 18). No evidence is provided in ICR whether the enactment of the amendments led to improved transparency and governance. The ICR notes one significant evolution: in December 2014 the controls for transfer pricing were restricted to international transactions while previously they covered both international and domestic cases. The project team observed that "the restriction of control to international transactions is a positive step in a low-capacity environment, allowing the administration to focus its efforts." Subsequent information was provided by the project team: "According to State Fiscal Service Data as of August 2017, from introduction of the law in 2014, 17 audits were completed with additional assessment for corporate tax of UAH 314 million, fines related to transfer pricing controls stood at UAH 160 million, and for financial years 2013-2016 taxpayers have corrected voluntarily taxable income by UAH 4.7 billion. In addition, the positive momentum created by this prior action helped start the comprehensive reform in international taxation resulting in current re-negotiations of tax treaties and Ukraine joining BEPS [Base Page 6 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) Erosion and Profit Shifting – an international collaborative effort to address tax avoidance strategies]." Objective 1 Conclusion. There were important program-supported achievements in establishing the legislative and institutional building blocks for enhanced governance. In some areas, such as investment management, there are indications that new procedure should lead to higher quality projects in alignment with strategic objectives. For the most part, however, evidence of specific outcomes was lacking, and implementation difficulties reflected the strong political opposition to some of the supported measures. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Strengthen the regulatory framework and reduce costs of doing business Rationale VAT Refunds. The DPO series supported several measures – publication of monthly data, reducing additional assessments, automating VAT refund procedures – to reduce delays in tax refunds and thus increase the liquidity of businesses. The ICR notes that the ratio of VAT refund claims older than 74 days was brought down from 153 percent in 2013 to 0.8 percent at end 2016, exceeding the target of 20 percent (ICR, page 20). Easing the regulatory burden on business. The DPO series supported the enactment of several pieces of legislation to facilitate doing business in Ukraine, inter alia reducing the number of permits from 143 in 2013 to 56 at the end of 2016, surpassing the target of 84 licenses by Q1 2017. The ICR argues that these eliminations have reduced the cost of doing business and presents compliance cost savings for the agribusiness. State Land Bank. The DPO operation supported policy actions to prevent the State Land Bank, which manages state-owned land and also acts as an agricultural development bank, from potentially misappropriating state (land) assets and supported measures to abolish the Bank. The measures included a draft amendment revoking the right of the SLB to count land towards its bank capital, thereby effectively eliminating the legal basis for its operations. Following the completion of the prior actions, the ICR notes that on July 26, 2016 the Ukrainian Central Bank declared the SLB insolvent (ICR, page 21). Objective 2 Conclusion. While it is reasonable to assume that the operation’s efforts in this reform area contributed to the results, the ICR does not present evidence of other, non-project factors that may have influenced these outcomes. This is important given the fact that other external partners – principally., the EU, the US and the IMF – were actively involved in the same reform areas. However, the task team leader stated Page 7 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) that most of these partners relied heavily on World Bank work and expertise in shaping their support. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Reform inefficient and inequitable utility subsidies while protecting the poor Rationale Reducing implicit energy subsidies. The DPO series supported several measures – i.e., tariff increases, strengthening the independence of the energy regulator, unbundling of Naftogaz. improving targeting – to reduce the high and poorly targeted energy subsidies. The ICR reports that gas and heating tariffs substantially increased – 450 and 250 percent respectively – which allowed the Ukrainian government to achieve full cost recovery at Naftogaz and district heating companies, one year ahead of the original target (ICR, page 23). The ICR also notes that the drop in energy prices also contributed to the achievement of the objective.　 While the formal objective in this area was to reform energy subsidies, the DPO series supported a more comprehensive reform package – including unbundling Naftogaz and strengthening the independence of the energy regulator – to increase structural efficiency in the energy sector. The ICR notes that progress with these reform broader measures has been weaker (ICR, page 23). For example, the second operation supported cabinet approval of a draft law to strengthen the independence of the national energy regulator. However, it took until November 2016 to get the law approved in Parliament and by March 2017 important elements of that law had not become operational. While the formal objective in this area was to reform energy subsidies, the DPO series supported a more comprehensive reform package – i.e., unbundling Naftogaz and strengthen the independence of the energy regulator – to structurally increase efficiency in the energy sector. The ICR notes that progress with these reform measures has been weaker (ICR, page 23). For example, the second operation supported cabinet approval of a draft law to strengthen the independence of the national energy regulator. However, it took until November 2016 to get the law approved in Parliament and by March 2017 essential elements of that law have not become operational. Strengthening social protection schemes. The first operation supported a social protection scheme that aimed to compensate the poorest 30 percent of the population for the gas and heating tariff increases. The ICR notes that this scheme had very low take-up rates due to parallel efforts in promoting another program, unfamiliarity of stakeholders and limited eligibility. The second operation shifted its focus, discontinued the original program and supported the more popular Housing and Utility Subsidy (HUS) scheme. Page 8 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) Enrollment in HUS increased from 1.3 million beneficiaries in 2013-14 to 6.6 million in 2016, significantly exceeding the enrollment target of 2 million households. Relying on a counterfactual scenario, the ICR provides evidence that the HUS scheme significantly reduced poverty, thereby reaching the objective of protecting the poor. However, the ICR also notes that further improvements in targeting of the program will be necessary as only 50 percent of the HUS budget reaches the poorest 40 percent of households. Objective 3 Conclusion. The available evidence suggests that the DPO series contributed to reforming utility subsidies while protecting the poor. On the other hand, the poor targeting of HUS and the discontinuation of initial social protection reforms point to efficiency concerns. Moreover, other, exogenous factors may have influenced these outcomes. However, the task team leader made clear that the World Bank, together with the IMF, were the key external partners in supporting Ukraine’s reform program. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Outcome The relevance of the program’s objectives and design is rated substantial. The program’s objectives were well aligned with the country’s development priorities and World Bank Group’s Country Partnership Framework. A logical causal link existed between policy actions and outcomes, and design took sufficient account of complementary factors that were important to achieve the intended outcomes. The efficacy of the program is assessed as substantial with regard to two out of the three objectives - – reducing the costs of doing business and reforming utility subsidies while protecting the poor. Despite some shortcomings, important outcomes were achieved in these areas. Concerning the third objective - promoting good governance, transparency and accountability in the public sector – there have been important achievements in building the legislative and institutional building blocks for enhanced governance. However, evidence of specific outcomes was lacking in most cases, and implementation difficulties reflected the strong political opposition to some of the supported measures. Overall, the shortcomings are considered to be minor, and outcome is assessed as satisfactory. a. Outcome Rating Satisfactory 6. Rationale for Risk to Development Outcome Rating The ICR highlights several risks that may negatively affect the realized outcomes: weak growth prospects and Page 9 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) macroeconomic vulnerabilities, potential escalation of the conflict in Eastern Ukraine, other geopolitical risks, backlash from vested interests, faded out reform momentum, low government capacity and reduced support of the wider development community. Concerning the latter, it is important to note that the IMF is continuing its support to Ukraine with an EFF program. a. Risk to Development Outcome Rating Substantial 7. Assessment of Bank Performance a. Quality-at-Entry The Bank team took advantage of the window of opportunity created by the Maiden Revolution (February, 2014) to support a critical and far reaching reform agenda. Under a tight time schedule – 1.5 months between DPL identification mission and Board approval – the team prepared the DPO series, cognizant of the geo- political and macroeconomic circumstances. Extensive policy dialogue was undertaken with different stakeholders. Preparation also benefited from substantial analytical work, coordination with multiple development partners, and complementarity with other operations. There were some minor shortcomings. The low take-up of the initial social protection scheme may reflect insufficient consultation, and there were weaknesses in M&E design. Quality-at-Entry Rating Satisfactory b. Quality of supervision The World Bank closely monitored reform progress with frequent updates to the Prime Minister, parallel technical assistance projects and continued coordination with development partners. When identified risks to the operation materialized, the Bank adjusted the design, recommended dropping prior actions and moved forward results indicators. Given a deterioration in the macroeconomic situation, the loan size of the second operation was doubled to 500 million USD. In hindsight, greater advantage might have been taken of the opportunity to intensify dialogue in support of the energy sector reform agenda, given the limited progress in that area. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 8. Assessment of Borrower Performance Page 10 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) a. Government Performance The ICR notes that following the Maidan Revolution, the government developed an ambitious reform program that, in part, relied on the DPO series to bring about change (ICR, page 30). The government implemented prior actions that directly affected the vested interests of influential politicians as well as tariff increases that were socially difficult. However, the reform momentum stalled in some areas, for example, the slow implementation of financial disclosure reforms, the unbundling of Naftogaz, the independence of the energy regulator. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance For this DPF the distinction between central government and implementing agency is minimal. Therefore, this review does not provide a separate rating implementing agency performance. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The program documents of both operations mention that only limited resources were being devoted to tracking progress and impact of reforms (Program Document P150313, page 27). Nevertheless, a set of indicators were developed with baselines and targets to assess the impact of the program. The M&E system mainly relied on a before-after analysis, which has obvious limitations (A time trend in a before-after analysis only has a causal interpretation if it is assumed that any observed changes over time are due to the program.) For a number of reform areas there appears to be a mismatch with the related indicators. For instance, the DPO series took several measures to improve the ease of doing business, going from food safety standards to deregulation, investor protection and business permits. The related indicator for those policy actions only measured the number of business permits required to start a business. Similarly, during the second operation the scope of the energy subsidy reform was widened, but the indicator only measured the financial deficit of Naftogaz. Finally, the indicator on public procurement inaccurately measured the reform efforts in that area. Monitoring and evaluation of the overall reform effort was to be coordinated by the Government under the National Reform Council of Ukraine. For three prior actions, indicators and targets were lacking: prior action 4 (DPO 2) on transfer pricing, and prior Page 11 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) actions 5 (DPO 1) and 7 (DPO 2), both on the State Land Bank. b. M&E Implementation There is little discussion in the ICR of M&E implementation. Given design limitations and inherent difficulties of estimating the impact of reform, the available evidence indicates that there was no methodologically sound analysis of data to estimate program impacts. However, the ICR does discuss the results of simulations of the energy reform on poverty. c. M&E Utilization The ICR does not report any utilization of the M&E system beyond monitoring program implementation. M&E Quality Rating Modest 10. Other Issues a. Environmental and Social Effects The DPO series supported measures to protect the poor from energy tariff increases. No environmental safeguard issues are discussed in the ICR. b. Fiduciary Compliance The ICR does not report on any fiduciary compliance issues. c. Unintended impacts (Positive or Negative) N/A d. Other N/A Page 12 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Substantial Substantial -- Outcome Bank Performance Satisfactory Satisfactory --- Given the stalled reform momentum in some areas, Borrower Performance Satisfactory Moderately Satisfactory Borrower performance is rated moderately satisfactory. Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons 　 A number of useful lessons are highlighted in the ICR. First, significant government commitment is paramount in achieving important reform outcomes. Second, the need to coordinate closely among development partners is critical in moving forward the reform agenda. Third, strong analytical work and deep engagement with government counterparts are important for successful policy lending. Fourth, flexibility in programmatic design allows the Authorities and the Bank to adapt the program to changing circumstances. 　 IEG adds two further lessons. First, patience enabled the Bank to seize a unique window of opportunity to support critical and far reaching reforms, after having refrained from policy lending when the timing was not right. Prior to the operations, the Bank maintained an open policy dialogue and produced significant analytical work, both of which were instrumental in　 enabling the program to be designed in a short time span. Second, deep-rooted reforms that go against vested interests are difficult to sustain once the reform momentum wanes. A thorough political economy analysis during or prior to preparation can provide useful insights in this respect. 13. Assessment Recommended? Yes Please explain Page 13 of 14 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPL 1(P150313) Yes. Given the significant risk to development outcome, indications of the stalled reform momentum, and the ambition to engage in far reaching reforms, an assessment is recommended. 14. Comments on Quality of ICR The ICR provides a detailed account and analysis of many important aspects of the program. It was clearly written, well structured, with plenty of background information. The ICR was also candid in pointing to limitations and reform difficulties – e.g., preparation time, vested interests, etc. These limitations however were not reflected in the ratings. The ICR would have benefited from a more elaborate counterfactual analysis, including a discussion on other, non-project factors that may have affected program outcomes. More attention could have been paid to the analytical work underpinning the DPO series. Greater detail on M&E implementation difficulties would have been useful. a. Quality of ICR Rating Substantial Page 14 of 14