Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) Report Number : ICRR0021656 1. Project Data Country Practice Area(Lead) Armenia Macroeconomics, Trade and Investment Programmatic DPL Planned Operations: 0 Approved Operations: 0 Operation ID Operation Name P127754 DPO 1 New Series L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-82940,IDA-53110 31-Mar-2014 72,000,000.00 Bank Approval Date Closing Date (Actual) 07-Nov-2013 31-Mar-2014 IBRD/IDA (USD) Co-financing (USD) Original Commitment 72,000,000.00 0.00 Revised Commitment 72,000,000.00 0.00 Actual 72,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Paul Holden Clay Wescott Malathi S. Jayawickrama IEGEC (Unit 1) PHPROJECTDATATBL Operation ID Operation Name P143040 ARMENIA DPO2 ( P143040 ) Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-82940,IBRD-84500,IDA-53110 31-Mar-2015 75,000,000.00 Bank Approval Date Closing Date (Actual) 12-Nov-2014 31-Mar-2015 IBRD/IDA (USD) Co-financing (USD) Original Commitment 75,000,000.00 0.00 Revised Commitment 75,000,000.00 0.00 Actual 75,000,000.00 0.00 PHPROJECTDATATBL Operation ID Operation Name P153234 ARMENIA DPO3 ( P153234 ) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-84500,IBRD-85700 30-Nov-2016 50,000,000.00 Bank Approval Date Closing Date (Actual) 09-Dec-2015 30-Nov-2016 IBRD/IDA (USD) Co-financing (USD) Original Commitment 50,000,000.00 0.00 Revised Commitment 50,000,000.00 0.00 Actual 50,000,000.00 0.00 PHPROJECTDATATBL Operation ID Operation Name P160100 Armenia DPF4 ( P160100 ) Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) L/C/TF Number(s) Closing Date (Original) Total Financing (USD) IBRD-82940,IBRD-85700,IBRD- 30-Nov-2017 50,000,000.00 86720,IDA-53110 Bank Approval Date Closing Date (Actual) 08-Dec-2016 30-Nov-2017 IBRD/IDA (USD) Co-financing (USD) Original Commitment 50,000,000.00 0.00 Revised Commitment 50,000,000.00 0.00 Actual 50,000,000.00 0.00 2. Program Objectives and Policy Areas a. Objectives The program development objective (PDO) of DPO1 was: “to support job creation through growth enhancing reforms” (Program Document (PD), p. v). The PDO of DPO2 is stated: “The series aims at: i) strengthening the business environment, ii) improving access to credit, iii) improving efficiency and transparency of the civil service, iv) expanding social protection, and v) improving fiscal space and the management of public infrastructure and environmental resources” (PD, p. v). The PDO of DPO3 is: “to support the government’s reform agenda in two key areas: (i) promoting fiscal, social, and environmental sustainability; and (ii) strengthening competitiveness” (PD, p. vi). There is an equivalent statement in DPO4. This Review uses the PDO statements of DPO3 and DPO4 along with the five sub-objectives of DPO2 as the basis for validating the ICR, because this combination gives an outcome-oriented statement of the subset of the government program supported by the series. Job creation is the overarching goal, and will be assessed under both areas. b. Pillars/Policy Areas There were two pillars, which were unchanged in all four operations (ICR. p. 6). They were: Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) • Strengthening Competitiveness o Improve the business environment through inspection agencies reform; o Improve access to credit through enhancing credit reporting, strengthened financial regulation, secured transactions reform, and improving the legal framework for bankruptcy and debt workout; o Civil Service Reform through improving the efficiency and transparency of public administration; o Increase the efficiency and transparency of customs; o Enhancing competition through improving air connectivity by stimulating competition in the air transport sector and improving regulation in air transport, railways and utilities; o Enhancing financial sector development by issuing enabling regulations for pension savings accounts, consolidating supervision, and enhancing transparency of ownership of financial institutions. In the second operation, this policy area was dropped due to its doubtful legality and public protest. • Enhancing Fiscal, Social and Environmental Sustainability, which included; o Tax Revenue Mobilization through reviewing tax exemptions and streamlining tax regulations as well as finalizing a new tax code; o Streamlining social safety nets through improving access to benefits; o Improving the financial and technical sustainability of key sectors to improve public service delivery and contribute to macroeconomic stability; o Mainstreaming environmental sustainability in the mining sector through adopting a new law for environmental impact assessments and operationalizing a one-stop shop for mining rights and regulating mining waste. c. Comments on Program Cost, Financing, and Dates DPO1 was approved, November 7, 2013, became effective on November 8, 2013 and closed on schedule on March 31, 2014. The operation consisted of a credit of US$41 million and a loan of US$31 million. The actual amount disbursed was US$72 million (ICR p. 8). DPO2 was approved on November 12, 2014 in the amount of US$75 million. It became effective on December 19, 2014. It closed on March 31, 2015 (ICR p.8). Note, the Data Sheet for Basic Information (ICR p. i) shows incorrect information for the DPO2 Disbursed Amount and the Key Dates for DPO2 (ICR p. ii) shows an incorrect closing date. DPO3 was approved on December 9, 2015 in the amount of US$50 million and became effective on December 15, 2015. It closed on schedule on November 30, 2016. The actual amount disbursed was US$50 million. (ICR p. 8) DPO4 was approved on December 8, 2016 in the amount of US$50 million. It became effective on December 23, 2016 and closed on schedule on November 30, 2017. The actual amount disbursed was US$50 million. (ICR p. 8) Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) 3. Relevance of Objectives & Design a. Relevance of Objectives The objectives were relevant to the country context. The 2017 Systematic Country Diagnostic (SCD) identified several issues under the rubric of Reducing Poverty and Boosting Shared Prosperity. One of these included strengthening environmental regulation, disaster risk management and natural resource management as well as strengthening social protection and improving basic services (SCD p. 45). The SCD also points out using several indicators that Armenia’s competitiveness in several areas, such as productivity was low and needed strengthening. This was translated in the Country Partnership Framework for FY19 - FY23 into 4 “Pathways”. Included under Pathway 4 was achieving sustainability by “build(ing) national resilience on multiple fronts” (p. 14). The other 3 Pathways included: • Rebalancing growth by seeking to expand markets through overcoming connectivity constraints; • Removing constraints to firms entering and growing markets to develop and vibrant and productive private sector; • Removing barriers to work and productivity improvements on the part of individuals. Overall the actions under the 4 DPOs promoted the objectives of the pathways. The objectives at the start of the series were aligned with the World Bank Country Partnership Strategy (CPS), FY2014-FY2017, the aims of which were to support the goals of the Government of Armenia in boosting shared prosperity and reducing poverty through accelerating growth and creating jobs. “The CPS places private sector led job creation at the center of the WBG’s response (CPS p. iv)”. This was to be achieved, the CPS stated, by first supporting improved competitiveness and job creation as well as increasing investment in job creating activities in areas outside Yerevan. Secondly, the CPS aimed to improve efficiency and targeting of social services in order to improve poverty and increase the impact of social services. The ICR (p. 17) states that all the operations in the series “supported the key objectives of the government’s Armenia Development Strategy for 2014-25”. There was a shift in strategy from job creation in the first 2 series to crisis mitigation in DPO3, which was understandably not mentioned in the earlier operations since the crisis was unanticipated. Nevertheless, the ICR also states (p. 17) that the “Program Objectives were not fully spelled out in all the operations of this series”. Furthermore, when the crisis hit, consideration could have been given to truncating the series at 2 operations and initiating a new series directed specifically at the crisis. At closing the objectives remained consistent with the FY14-FY17 CPS. The subsequent Country Partnership Framework was only introduced in FY19. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) Rating Substantial b. Relevance of Design Corruption, lack of access to finance and government inefficiency were the three most binding constraints that were identified by businesses in surveys conducted for the 2013-2014 Global Competitiveness Report. However, the Armenia Enterprise Survey 2013 cites tax rates and tax administration and political instability as the largest obstacles affecting investment and entrepreneurship (https://www.enterprisesurveys.org/content/dam/enterprisesurveys/documents/country-profiles/Armenia- 2013.pdf). The implicit causal chain underlying the design of the four-program series was based on the hypothesis that reducing the constraints to doing business in Armenia would stimulate job growth, reduce unemployment, create investment and entrepreneurship (ICR p. 1). This would also result in existing firms expanding, and new businesses starting. It would also integrate the economy more closely with international markets. This is consistent with research that has sought to explore the relationship between the investment climate, firm performance and employment growth (see, for example, Batra G. and Andrew Stone, 2008, Investment Climate, Capabilities and Firm Performance: Evidence from the World Business Environment Survey, OECD Journal, General Papers; and Rahman, Aminur, 2014, Investment Climate Reform and Job Creation in Developing Countries: What Do We Know and What Should We Do?, Policy Research Working Paper, 7025, World Bank). The link between employment creation and the actions under the series is somewhat tenuous, however, since job growth arises from both demand side and supply side factors. This is clearly delineated in The Supplement to the IMF Article IV consultation for 2019, which states (p. 49) “Declining employment during recent years of relative macroeconomic stability suggests that demand stimulating policies alone are unlikely to address high unemployment”. In the first two programs under the series, the focus was on the investment climate and social protection. The reforms that the series addressed focused on reducing corruption through simplifying inspection processes, improving access to finance, enhancing the efficiency of the civil service, improving infrastructure, and improving competition in the transport sector. It also aimed at increasing tax revenue to promote fiscal sustainability, strengthen the social safety net and improve environmental sustainability. While some of these policy goals had a direct bearing on the investment climate, others appeared to be no more than tangentially related, and it is hard to avoid the conclusion that these constituted a “grab-bag” of reforms and that a narrower focus might have been more effective. In addition, it is difficult to discern a specific connection with job creation in many of these reforms, particularly those relating to the social safety net. The macroeconomic framework was presumably addressed through measures to increase tax revenue by reforming the tax code, but the reforms are not convincing in this area because they are so narrowly focused. Furthermore, the deterioration of macroeconomic conditions as the series progressed indicates that the first two series, at least, did not appropriately address the macroeconomic framework – see following paragraph. Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) An additional issue arose because of the evolution of events as the series progressed. Over the course of the series, there were two reshufflings of the government, external conditions deteriorated, and macroeconomic risks increased. Public protests also played a part in program changes. In 2015 the Government attempted to increase electricity tariffs in order to clear the accumulated losses of the electricity generator but this engendered widespread public demonstrations. These events resulted in the addition of a fourth DPO in the series as well as changes to the earlier DPO operations. Because of numerous changes of indicators, the revision of indicators, the use of pillars as PDOs and the disconnect between triggers and prior actions in DPO2 and DPO3 (discussed in detail in Section 7a), the relevance of design is rated modest. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Promoting Fiscal, Social and Environmental Sustainability Rationale This PDO had 4 sub-objectives. All four DPOs had prior actions under this objective. (i) Fiscal Sustainability Under DPO1, fiscal sustainability was promoted through the adoption of a decree that outlined guidelines and principles for examining tax and customs legislation, and in particular a review of tax exemptions and duties in order to promote revenue collection. In addition, a medium-term strategy was developed to deliver amendments to tax and customs legislation. Under DPO2 the prior action required improving tax mobilization through a draft law submitted to Parliament that rationalized VAT exemptions for financial brokerage services. Under DPO2, this was extended to creating fiscal space through removing some tax expenditures that would allow for the improvement in drinking water, irrigation and roads. Under DPO3 the prior action required approval of the draft Unified Tax Code that was published on the governments website and under DPO4 a unifying tax code would be enacted to improve the efficiency transparency and equity of the tax system as well as strengthening tax administration. The initial results indicator for this PDO under PD1 was the ratio of tax to GDP. The base (2012) was 21.4 per cent, with a target value by 2015 of 23.5 per cent under DPO1. By 2017, the ratio was 21.3 per cent, although this target was dropped in DPO2, added back in DPO3 and dropped in DPO4. In DPO4, a new indicator was added, namely the ratio of excise taxes on tobacco and alcohol, which had a value of 0.9 per Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) cent in 2012, 0.72 in 2015 and a target of 0.88 per cent in 2017. This was met, with the actual being 1.1 per cent in 2017. An additional indicator introduced in DPO2 was to publish the total amount of tax exemptions. The indicator was met, with the amount being published in the state budget of 2015. In DPO3, an additional indicator was introduced on the number of tax exemptions, this was not met as no data were published. (ii) Social Sustainability Under this sub-objective there were prior actions in DP1-DP3. Under DP1, the prior action required improved targeting accuracy and coverage of the Family Benefits program; under DP2 the prior action was the extension of coverage of health benefits and under DP3, the prior action was the passage of a law on Social that integrated the provision of Social Services. There were three outcome indicators under this objective. The indicators and outcomes were: • The number of households covered by the family program. The 2012 base was 96,309 with a target of 104,000 by 2015 and 105,000 by 2017 under DPO 4. The actual number at the end of 2016 was 105,150; • The share of poor households covered by the family benefit program.; The 2012 base was 21 percent with a target of 27 percent by 2016. The actual number at the end of 2016 was 24.3 percent; • The number of consolidated social service access points. The baseline in 2012 was zero. The target number under PDO1 by the end of 2016 was 30. The actual number achieved by the end of 2018 was 26. (iii) Improving infrastructure and public service delivery in key sectors These sectors included strengthening drinking water supply, irrigation and roads in a way that would also contribute to macroeconomic stability and resilience. Under this sub-objective, there were prior actions in the second through fourth operations. Under DPO2, an action plan for reforming the supply of drinking water through the creation of a nationwide lease contract and the issuing of tenders for the same. Under DPO3, the prior action was to increase the electricity tariff to make electricity generation more sustainable. Under DPO4, the prior action was to approve a strategy for securing the financial sustainability of the agriculture irrigation sector. There were five results indicators under this sub-objective. The indicators and outcomes were: • The percentage of the road network that was routinely maintained. In 2012 the base was zero with a target by 2016 or more than 60 per cent. This target was met in 2017; • The number of water user associations setting fees according to their business plans. The base in 2012 was zero with a goal of more than five by 2016. This indicator appears to have been incorrectly chosen because the water user associations have financial plans - they do not have business plans (ICR p.36); • A reduction in irrigation subsidies – with a target of five percent lower subsidies by 2017, which was met although the ICR (p. 36) states that the action plan for improving the financial sustainability of irrigation was not implemented because of the adoption of a protocol decree that was non-binding; • The number of self-financing water user associations from a base was zero in 2015 with a target of five by 2017 under DPO4. This was not met – by 2017 there were no self-financing water user associations; • A decline in the accumulated net losses of ENA, the electricity generator - in 2015 ENA was privatized and Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) tariffs were raised which sharply reduced the losses. The ICR (p.37) reports that the electricity regulator is considered to be one of the best in the region. (iv) Environmental Sustainability Under this sub-objective, there were prior actions under all 4 of the DPO series. For DPO1, the prior action was that the government would develop and adopt a new draft law on undertaking environmental impact assessments that reflected international best practices. Under DP2, the prior actions required the government to strengthen the management of natural resources by Parliament approving the draft law developed under DP1. Further, the government would approve a one stop shop for mining rights that would respect environmental and social guidelines. Under DP3, the government would amend on law on waste management to improve the regulation of mining waste. Under DP4, the government would amend the Mining Code to be consistent with the Law on Waste Management. There were four result indicators under the sub- objective. The indicators and outcomes were: • The number of mining licenses issued in accordance with the new environmental law. The 2012 base was zero with a target under DPO1 of more than one. By 2017, 48 licenses had been issued; • The number of mining license applications issued through the One stop Shop. The base was zero with a target of more than one by 2016. By 2017, 48 licenses had been issued; • The number of mining licenses issued in accordance with the amended Law on Waste Management and the provisions in the Mining Code. The base was zero with a target by 2016 of more than one. Under DP04 the target was that all licenses will be issued in terms of these provisions. By 2016, 55 licenses had been issued. However, the results indicators do not relate adequately to environmental sustainability because of their excessive focus on the issuing of licenses. The issuing of mining licenses, even though they might have been in accord with the new Law does not indicate the extent to which the law was being followed – implementation of the law is the key to ensuring no environmental degradation. Because many of the indicators fail to demonstrate that the development sub-objectives were being achieved, the rating under this sub-objective is therefore modest. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Strengthening Competitiveness Rationale Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) This PDO had 5 sub-objectives; improving the business environment; strengthening the civil service; improving trade facilitation and connectivity; enhancing access to credit and improving regulation of the financial sector. In the first operation, pension reform had been included as a sixth sub-component but was dropped in the subsequent DPOs due to legality issues as well as public protests. (i) Strengthening the Business Environment Under this sub-objective, there were prior actions under DPO1-DPO4 that revolved around the reform of the inspections and licensing systems, which the government and the World Bank viewed as being open to corruption through excessive inspections on businesses. Under DPO1, the prior action was that the government merge the Labor Inspectorate and the Anti-Epidemiology Inspectorate in order to reduce the number of business inspections. Under DPO2, this was taken further through the prior actions of adopting a reform strategy for the optimization of the inspection agencies system, that would make it more transparent and accountable. Further, the prior action required an electronic system for creating e-government applications be established. Under DPO3, the prior action required that inspections procedures be strengthened through the passage of a law governing inspection agencies, that along with the accompanying regulations would facilitate the provision of electronic information on inspections. In addition, the prior actions required the introduction of a streamlined licensing system. There were five results indicators under this sub-objective that revolved around inspections and licensing. The indicators and outcomes were: • The number of inspection agencies: The 2012 base was 18 with the target by 2017 of 9.. By 2017 the actual number remained at 15, but by 2018 it had fallen to six as a result of the consolidation of inspection bodies • Management boards would be functioning for all inspection agencies: The base in 2012 was zero with the target of nine by 2017. By 2017 the actual number was four but by 2018 it had risen to six with management boards functioning for all the consolidated inspection agencies; • The number of tax inspections: The base for this indicator (in 2013) was 1470. The target was a 20% reduction by DPO3, but by 2017 the actual number was 1440, with hardly any change having been achieved; • A reduction in licensing requirements: The base for the number of economic activities subject to licensing requirements was 95, with a target under DPO3 of 81. The actual number achieved was 85. • The number of applications approved through simple notification mechanism rather than specific licensing: The base for 2012 was zero with the target in 2017 of 35 under DPO4. The actual result in 2017 was seven. The ICR (p. 33) indicated that in the first half of 2018, 17 applications were approved through the notification mechanism, suggesting that the system was being increasingly used. All of these measures are, however, inputs to an anti-corruption strategy, with their effectiveness and enforcement being key missing issues. It is not possible to determine if these measures reduced corruption, but the evidence for them having done so is thin. The IMF Article IV consultation paper for 2019 states (p. 11) “Importantly, the still-weak business climate and corruption constrain the economy’s capacity to grow sufficiently rapidly to tangibly reduce poverty and unemployment.” Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) Notably absent were references to employment, even though business environment reform was intended to stimulate employment growth. Over the life of the series, employment declined from 1.164 million at the end of 2013 to 1.142 million at the end of 2018., although there was a significant uptick in employment growth between 2017 and 2018 (source: https://www.armstat.am/file/article/armenia_2019_4.pdf and https://www.armstat.am/file/article/trud_18_4.1.pdf). Since many factors impinge on employment, attribution would have been difficult, however. (ii) Enhancing Access to Credit Prior actions under this sub-objective were required under all of the DPOs. Under DPO1, the prior action required that the Central Bank implement measures to enhance the operation of the credit registry, clarify the role of the Financial System Stability and Development Consumer Protection Unit in supervising the registry, and strengthen and clarify the role of a number of organizations associated with the protection of consumer rights. Under DPO2, the prior action consisted of adopting legislative changes that would allow the use of moveable assets as collateral for lenders. This required the amendment number of laws governing the operation of the financial system. Under DPO3, the prior action required the enactment of a new Law on Secured Transactions to further strengthen the ability of borrowers to pledge movable property as collateral for obtaining loans. Under DPO 4, the prior action required the enactment of amendments to the bankruptcy law. The indicators and outcomes related to this sub- objective were: • The number of credit reports issued on individuals. The 2012 base was 75,000 with a target of 150,000 by the end of DPO2, and a target of 4.5 million under DPO4. The actual number achieved by 2017 was 5,974,386. • The number of consolidated audits carried out by the Central Bank of Armenia: the base for this indicator was zero. It is important to note that until 2016, consolidated audits carried out only in special cases. Since 2017 banks are required to provide information each year. The actual number of audits achieved by the end of 2017 was four against a target of at least one. • The time required for resolving insolvency as measured by the Doing Business Report. The 2012 base was 1.9 years with a target at the end of 2016 under DPO2 of 1.5 years. The ICR (p. 33) indicated that the DBI information showed that at the end of 2017 there had been no change compared to the base year. Nevertheless, the ICR states (p. 33) that the ICR mission had been told that in practice the process takes 6 – 12 months. This illustrates the caution with which the DBI indicators must be treated, and the doubtful validity of using them for providing indicator data (See Paul Holden and Alma Pekmezovic, 2019, The World Bank Doing Business Indicators, Inconsistent, Unreliable and Inaccurate, mimeo. • The index of strength of the insolvency framework in the DBI's: this indicator was introduced in DPO's three and four. The base year was 2015 with an index score of nine and a target at the end of 2018 of 12 the ICR reports that at the end of 2017 the actual index score was 7.5 out of 16. These indicators do not measure whether access to credit improved, nor whether the new legal bankruptcy framework achieved greater efficiency. In addition, there is no mention of the extent to which they impact job creation. (iii) Strengthening the Civil Service Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) Prior actions under this sub-component were required for DPO1, DPO2 and DPO4, which related to civil service reform that would strengthen recruitment and promotion (DPO1), enhance the performance evaluation system (DPO2), and the introduction of regulations to further improve civil service evaluations and performance bonuses. There were three outcome indicators under this subcomponent. The indicators and results outcomes were: • The share of eligible external applicants during competitions for professional civil service positions. The base for this indicator in 2012 was zero. The 2016 target under DPO's 1 and 2 was greater than 10 percent while the reported achievement for 2017 was 85%. However, the ICR reports that the data are inconsistent relative to the numbers in the program documents and therefore cannot be evaluated. In any event, this indicator is an input and provides no information regarding whether the efficiency of the civil service had in fact improved. • The average value and standard deviations of performance appraisal scores of civil servants. The 2015 base for this indicator was 95% and three, while the 2017 target under DPO four was 95% and five respectively. The actual values achieved were 91.2% and 5.5. This indicator also says nothing about the output performance of the civil service. • The number of electronic identifications and certificates issued. the base for this indicator in 2012 was 26,000 with a target for 2016 under DPO's 1 through 3 was 250,000. for DPO4 the 2017 target was 900,000. The actual 2017 number issued was 916,252. Again, the relationship to the over-arching objective of job creation is missing. (iv) Enhancing Trade Facilitation and Connectivity Under this sub-component, prior actions were required under DPO1-DPO3. Under DPO1, the prior action consisted of adopting a new aviation policy that would increase transparency and competition in the aviation sector. This was taken forward in the prior action under DPO2, which specified a new institutional structure for the oversight of the aviation sector. Under DPO3, the prior action specified that the law governing aviation be amended to define procedures for licensing air operators and certain dictation as well as establishing criteria for Aviation Council. There were two output indicators for the subcomponent. They were, together with outcomes: • The first was the average time taken to complete the processing of goods for imports and exports. The base was unknown although under DPO1 the target was a 10% decline in processing time as evidenced by the doing business indicators. The ICR (P. 34) indicates that in 2013 it was 17 days and by 2018 had declined to 24 hours. However, the ICR also points out that progress cannot be measured, firstly because the base was unknown and secondly because the Doing Business methodology was changed although the ICR also indicates that there had been an improvement. • The second indicator was the percent of non-energy imports processed through the "single window approach". The base in 2012 was 20% with a target under DPO's 2 through 4 of 45%. By 2017 the number was 85.1% but the ICR indicates that this cannot be evaluated because of data inconsistencies. Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) (v) Enhancing Competition in the Air Sector Under this sub-component, prior actions were directed at liberalizing the aviation sector. There were policy actions required under the first three operations. Under DPO1, a new aviation policy was adopted as a result of the completion of a study on policy options for the liberalization of the air transport sector. Under DPO2, the prior actions related to the creation of a new institutional structure for aviation that assigned the responsibility for policy making to the Ministry of Economy and the responsibility for implementation to the General Department of Civil Aviation, which would decide on licensing and air service agreements as recommended by a multi-stakeholder council. Under DPO3, the prior action related to the amendment of the Aviation Law and the issuing of regulations related to licensing, certification and the establishment of an Aviation Council. The indicators and outcomes under this subcomponent were: • The number of air travel markets served from Yerevan airport. The 2013 base was 33. The target for 2016 under DPO1 was more than 33. The actual numbers achieved had declined to 29 in 2017. • The number of airlines serving Yerevan airport was to rise from a base of 36 in 2014 to 40 in 2016 under DP2. By 2017 the number of airlines serving the airport had declined to 27. • The number of passengers arriving in Armenia by air was to rise by 25% by 2018 from a 2012 base of 1,691,000. The actual number achieved by 2017 was 2,553,914. Many of the indicators fail to demonstrate that the development sub-objectives were being achieved and a connection to job creation is missing, with neither total employment nor the unemployment rate being selected as indicators. The ICR (p. 18) suggests that “employment is typically a lagging indicator” and that the uptick in the employment rate in the first half of 2018 demonstrates that the program measures were successful in creating jobs. Recent data go some way to supporting this contention. In October 2019, total employment amounted to 1.172 million compared to 1.142 million at the end of 2018 and 1.073 million at the end of 2015. Nevertheless, the number of employed is still no more than it was at the end of 2012. Furthermore, the statistics could well contain a lot of “noise. At the end of May 2019, the unemployment rate was 21.9 per cent, compared with 16.9 per cent at the end of December 2018. However, in June it declined by 4.2 percentage points to 17.7 per cent about the same as it was at the end of 2014 suggesting some underlying unreliability of the data. The rating under this sub-objective is therefore modest. Rating Modest PHREVDELTBL PHREVISEDTBL Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) 5. Outcome The relevance of the objectives was rated substantial because of its consistency with the Country Partnership Strategy FY14-FY17. While the ICR pointed out that the objectives of the series were not always made explicit, their relevance was sufficiently germane to justify the rating. The relevance of design was rated modest on the grounds that the causal chain was often not fully specified, the reforms were overly general and did not always map into objectives, and that target indicators were poorly designed, which changed often, and at times could not be evaluated. Furthermore, the relationship with the overarching objective of job creation was unclear. Efficacy under the first objective – promoting fiscal, social and environmental sustainability – was rated modest because of the failure to achieve targets and because some of the indicators measured inputs rather than outcomes. Efficacy under the second objective was also rated modest on the grounds that for four of the five subcomponents the indicators did not demonstrate improvements in outcomes. a. Outcome Rating Moderately Unsatisfactory 6. Rationale for Risk to Development Outcome Rating The ICR (pp. 22-23) lists four risks to development outcome; macroeconomic risk, political and governance risks, risks from weak institutional capacity to implement and sustain reforms as well as geopolitical risks. The 2019 IMF Article IV Report indicates that progress has been made in attenuating macro-economic challenges, which will be helped by the recently announced Standby Agreement, it nevertheless reiterates the significant challenges that continue to confront the Armenian economy. The change in government that occurred in May 2018 (ICR P.23) has resulted in stronger commitment to reform but political risk also remains high. Undertaking reform is further complicated by the weak institutional capacity to implement and maintain reform momentum on the part of middle level civil servants. In addition, geopolitical tensions continue to constrain the economy particularly those related to border access, with Armenia having land access only through Iran and Georgia. a. Risk to Development Outcome Rating Substantial 7. Assessment of Bank Performance a. Quality-at-Entry The DPO series was requested by the Government of Armenia to support its development strategy. The series was, overall, very broadly consistent with components of the strategy and were directed at some of the issues that had formed part of the reform agenda, including tax reform and reforms to the aviation sector. However, there were several shortcomings in Quality at Entry. The program document for the first Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) operation under the series stated (p. 1) that the new series built upon measures under the previous three- year 2009 to 2012 DPO series. There were significant shortcomings in program design, particularly as they related to poorly defined Development Objectives, changes in prior actions and priorities throughout the series, confusion between inputs and outcomes in the design of indicators, a failure to translate triggers into prior actions throughout all of the series, substantial changes in results indicators and an overestimation of the capacity of government counterparts. Examples include only 5 of the 11 DPO1 triggers becoming prior actions in DPO2 and in DPO2, only one of the 12 triggers became prior actions in DPO3. The ICR (p. 26) states “Review of internal documents pertaining to the preparation of the DPOs suggests that the Bank teams could have been more responsive to comments raised at various stages of the process.” The ICR also states (p. 5) that there was insufficient engagement with civil society while the program was being designed. Bank performance at entry is therefore rated unsatisfactory. Quality-at-Entry Rating Unsatisfactory b. Quality of supervision The ICR (p. 24) reports that during the preparation of each follow-up DPO, the World Bank teams supervised the implementation of the program for the preceding operation. It also reports that there was close coordination with the office of the Deputy Minister of Finance for the first two operations and with the Deputy Prime Minister for the subsequent two operations. The ICR does not discuss whether the supervisory process was related to the substantial changes in the indicators that occurred as the four operations progressed. According to the ICR, Armenian counterparts acknowledged the usefulness of the technical advice provided by the project teams during the implementation of the program and their responsiveness to government needs. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 8. Assessment of Borrower Performance a. Government Performance The ICR (p. 25) points out that the government was strongly committed to achieving the development objective against a background of domestic and external challenges that occurred over the period of the programs. Other positive aspects of government performance were: Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) • The implementation of the necessary legal and regulatory reforms that were part of the program; • The extensive consultations that took place during the of development of and revisions to the Unified Tax Code; • The engagement of senior government officials with the program. Nevertheless, as the ICR states (p. 25) there were delays in implementation and shortfalls in meeting the program targets. The reform of the pension system had to be postponed and there was not progress on reducing subsidies to irrigation. Furthermore, further work is needed to improve the tax code. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance N/A Implementing Agency Performance Rating Not Rated Overall Borrower Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design For DPO1 and DPO2, the Ministry of Finance was responsible for collecting data and monitoring progress under the auspices of the Deputy Minister of Finance. Under DPO3 and DPO4, the Deputy Prime Minister was overall responsible for monitoring and evaluation. The framework for monitoring progress was established by a Prime Minister’s decree promulgated in 2009. Furthermore, indicators related to achieving the objectives were revised or changed several times over the series. The ICR (p. 5) states “Changes to the results indicators reflected significant adjustments to the program”, which occurred in response to changing conditions. Furthermore, given the focus on job creation, particularly in DPO1 and DPO2, a total employment indicator could have been included. While it would have been difficult to attribute developments in employment to actions under the program, this indicator would have provided information on labor market challenges in Armenia. The ICR also points out that target indicators were initially poorly designed and were not sufficiently discussed with civil society and did not allow for lack of implementation capacity with Armenian counterparts. Furthermore, at times changes to results indicators and prior actions were made (ICR p. 17), without in-depth justification. Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) Indicators for the initial operations under the series were dropped and new ones added, so that the evolution of the success of the program series was difficult to follow. The ICR states (p. 4) “Results Indicators (RIs) were revised, dropped and created numerous times after the first operation”. Table 1 shows changes from one operation to the next. Furthermore, RIs changed both in terms of their definition as well as their target values.” Changes to Results Indicators in the Four Operations Operation Number of Number Dropped in the Number Added Number RIs Following DPO in the of RIs Following DPO Revised DPO1 19 8 4 DPO2 15 7 7 DPO3 15 3 3 4 DPO4 15 N/A N/A 6 The ICR (p. 19) states “it is hard to ascertain the actual reasons behind most changes.” Furthermore, some obvious indicators were not used. For example, changes in access to credit could have been monitored using the number of security interests registered in the filing archive associated with the secured transactions reform because as a rough approximation, each secured loan would have had associated with it an entry in the filing archive. b. M&E Implementation The ICR contains little information on M&E implementation beyond stating (p. 17) that “The Deputy Prime Minister monitored achievement of objectives periodically” c. M&E Utilization The same statement as above applies to M&E utilization M&E Quality Rating Modest 10. Other Issues a. Environmental and Social Effects The ICR (p. 21) states that the program had a positive impact on the environment through the amendment of the Mining Code, particularly the sections governing waste management and processing, and through the bankruptcy code amendments that incorporated environment liabilities in the restructuring of insolvent companies. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) With regard to social effects, the extension of the coverage of health benefits and the expansion of access to social services had a positive impact on social development. b. Fiduciary Compliance The ICR contains no reference to fiduciary issues c. Unintended impacts (Positive or Negative) N/A d. Other N/A 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Only modest achievements Moderately Moderately Outcome under efficacy and modest Satisfactory Unsatisfactory design Risk to Development Substantial Substantial --- Outcome Poor quality at entry and Moderately Moderately Bank Performance numerous revisions to the Satisfactory Unsatisfactory program Moderately Borrower Performance Moderately Satisfactory --- Satisfactory The ICR described well the Quality of ICR Substantial complexities of the program Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 12. Lessons Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) The following lessons are taken from the ICR with some modification of language by IEG. A series of 4 DPOs can often be excessive. Changing conditions could necessitate significant changes to the program, which complicates design, implementation and evaluation. In Armenia, by the time the fourth DPO commenced circumstances had changed substantially compared with the first DPO. Armenian authorities indicated that two DPO series that each had 2 programs could have improved performance. Excessively broad PDOs complicate design, greatly increase the difficulty of effective implementation, complicate monitoring and makes effective evaluation highly problematic. In the case of Armenia, the numerous changes the failure to translate triggers in a preceding DPO into prior actions in subsequent DPOs complicated achieving the targets of the program. Pressures to disperse can compromise the effective implementation of a programmatic series. Furthermore, it can also lead to shortcuts being taken to comply with prior actions. This results in less effective or delayed reforms. In the case of Armenia, the pressure to disperse resulted in a tax law that contains many weaknesses, and laws and regulations in the mining sector that were not fully effective. IEG adds the following lessons. Undertaking numerous changes to results indicators and modifying targets greatly complicates both program implementation and evaluation. In the case of Armenia, widespread changes in results indicators made the effective monitoring and evaluation of the program extremely difficult. When designing a Monitoring and Evaluation Framework, it is important to ensure that the indicators relate to outcomes and not to inputs. In the case of the Armenia DPO series, numerous indicators measured inputs. If a DPO series has an overarching objective, including indicators that provide information on whether it is being achieved strengthens the evaluation of its success. In the case of Armenia, job creation was the overarching objective, but there were no indicators containing employment data 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR rated achievement of objectives in terms of pillars rather than development objectives. However, the ICR analyzes and describes well the complex series of operations in a clear manner and given that this was a series of four operations is relatively concise. It describes well the background to the series and the challenges confronting Armenia as circumstances involved. The problems with the preparation of the series, how they evolved, and the issues related to the negative implementation of the program are documented. However, many of these references to the problems are scattered throughout the document and are at times difficult to Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review DPO 1 New Series(P127754) connect. While the ICR is frank about many of the program failures, it fails to take these into account in its ratings of the different aspects of the program in terms of which it is overly generous in some areas. a. Quality of ICR Rating Substantial Page 20 of 20