ARMENIA Achievements and Challenges in Improving Health Care Utilization A Multiproject Evaluation of the World Bank Support to the Health System Modernization (2004-2016) Report No. 134584 MARCH 20, 2019 © 2019 International Bank for Reconstruction This work is a product of the staff of The World RIGHTS AND PERMISSIONS and Development / The World Bank Bank with external contributions. The findings, The material in this work is subject to copyright. 1818 H Street NW interpretations, and conclusions expressed in Because The World Bank encourages Washington DC 20433 this work do not necessarily reflect the views of dissemination of its knowledge, this work may be Telephone: 202-473-1000 The World Bank, its Board of Executive reproduced, in whole or in part, for Internet: www.worldbank.org Directors, or the governments they represent. noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: The World Bank does not guarantee the World Bank. 2019. Armenia—Achievements accuracy of the data included in this work. The Any queries on rights and licenses, including and Challenges in Improving Health Care boundaries, colors, denominations, and other subsidiary rights, should be addressed to Utilization: A Multi-Project Evaluation of the information shown on any map in this work do World Bank Publications, The World Bank World Bank Support to the Health System not imply any judgment on the part of The Group, 1818 H Street NW, Washington, DC Modernization (2004-2016). Independent World Bank concerning the legal status of any 20433, USA; fax: 202-522-2625; e-mail: Evaluation Group, Project Performance territory or the endorsement or acceptance of pubrights@worldbank.org. Assessment Report 134584. Washington, DC: such boundaries. World Bank. Report No.: 134584 PROJECT PERFORMANCE ASSESSMENT REPORT ARMENIA HEALTH SYSTEM MODERNIZATION PROJECT (APL I) IN SUPPORT OF THE FIRST PHASE OF THE HEALTH SECTOR REFORM PROGRAM IDA CREDIT NO. 39200 HEALTH SYSTEM MODERNIZATION PROJECT (APL II) IN SUPPORT OF THE SECOND PHASE OF THE HEALTH SECTOR REFORM PROGRAM IDA CREDIT NO 42670 ADDITIONAL FINANCING FOR THE HEALTH SYSTEM MODERNIZATION PROJECT (APL II) IBRD LOAN NO.79870 March 20, 2019 Human Development and Economic Management Independent Evaluation Group Currency Equivalents (annual averages) Currency Unit = Armenian dram (dram) 2004 $1.00 dram 533.43 2011 $1.00 dram 372.50 2005 $1.00 dram 457.70 2012 $1.00 dram 401.76 2006 $1.00 dram 416.04 2013 $1.00 dram 409.62 2007 $1.00 dram 342.08 2014 $1.00 dram 415.91 2008 $1.00 dram 305.96 2015 $1.00 dram 477.91 2009 $1.00 dram 363.28 2016 $1.00 dram 480.48 2010 $1.00 dram 373.66 Abbreviations ALOS average length of stay IEG Independent Evaluation Group APL adaptable program loan IT Information technology BBP basic benefit package NHA National Health Account CBA cost-benefit analysis PHC primary health care EMP Environmental Management Plan PHRD Policy and Human Resources GDP Gross domestic product Development HPIU Health Project Implementing Unit PPAR Project Performance Assessment HSPA Health Sector Performance Report Assessments SHA State Health Agency HWM health waste management USAID United States Agency for ICR Implementation Completion and International Development Results Report WHO World Health Organization IDA International Development Association All dollar amounts are U.S. dollars unless otherwise indicated. Fiscal Year Government: January 1–December 31 Director-General, Independent Evaluation Ms. Alison Evans Director, Human Development and Economic Management Mr. Auguste T. Kouame Manager, Corporate and Human Development Ms. Emanuela Di Gropello Task Manager Ms. Mercedes Vellez ii Contents Preface ................................................................................................................................................................vii Summary ............................................................................................................................................................. ix 1. Background and Context ..........................................................................................................................1 2. Health System Modernization Project (APL I) ...................................................................................3 Relevance of the Objectives and Design..............................................................................................3 Objectives ..................................................................................................................................................................... 3 Relevance of the Objectives ...................................................................................................................................4 Design ............................................................................................................................................................................4 Relevance of Design .................................................................................................................................................. 6 Implementation ...........................................................................................................................................11 Key Dates ..................................................................................................................................................................... 11 Planned versus Actual Expenditure by Component...................................................................................... 11 Implementation Experience ................................................................................................................................... 11 Achievement of the Objectives ............................................................................................................. 13 Objective 1: Expanding Access to Quality Primary Health Care ............................................................... 13 Objective 2: Improving Quality and Efficiency of Selected Hospital Networks .................................. 15 Objective 3: Laying Groundwork for Effective Health Sector Policy Making and Monitoring (Governance) ............................................................................................................................................................. 17 Efficiency ....................................................................................................................................................... 19 Outcome ..................................................................................................................................................................... 21 Risk to Development Outcome ........................................................................................................................... 21 Bank Performance .................................................................................................................................................... 21 Borrower Performance .......................................................................................................................................... 22 Monitoring and Evaluation .................................................................................................................................. 23 3. Health System Modernization Project (APL II) ............................................................................... 24 Relevance of the Objectives and Design........................................................................................... 24 Objectives .................................................................................................................................................................. 24 Relevance of the Objectives ................................................................................................................................ 24 Design ......................................................................................................................................................................... 25 Relevance of Design ............................................................................................................................................... 26 Implementation ......................................................................................................................................... 28 Key Dates ................................................................................................................................................................... 28 iii Planned versus Actual Expenditure by Component.................................................................................... 28 Implementation Experience ................................................................................................................................. 29 Achievement of the Objectives ............................................................................................................ 29 Objective 1: Provide More Accessible Health Care Services to the Population ................................. 29 Objective 2: Provide More Quality Health Care Services to the Population ....................................... 31 Objective 3: Provide More Sustainable Health Care Services to the Population .............................. 33 Objective 4: Strengthen Ministry of Health Capacity for More Effective System Governance .... 34 Efficiency ...................................................................................................................................................... 35 Ratings .......................................................................................................................................................... 36 Outcome .................................................................................................................................................................... 36 Risk to the Development Outcome .................................................................................................................. 36 Bank Performance ................................................................................................................................................... 37 Borrower Performance .......................................................................................................................................... 37 Monitoring and Evaluation .................................................................................................................................. 38 4. APL Program Achievements and Challenges ................................................................................. 39 5. Lessons.......................................................................................................................................................... 41 Bibliography..................................................................................................................................................... 44 Figures Figure 2.1. Health System Modernization: Theory of Change.............................................................9 Appendixes Appendix A. Basic Data Sheet ................................................................................................................... 46 Appendix B. Methodology ......................................................................................................................... 52 Appendix C. Economic, Social, and Health Indicators ..................................................................... 56 Appendix D. List of Persons Met.............................................................................................................. 81 This report was prepared by Mercedes Vellez, who assessed the project in October 2018. The report received contributions from Carla Pazce and was peer reviewed by Judyth L. Twigg and panel reviewed by Soniya Carvalho. Aline Dukuze and Yezena Yimer provided administrative support. iv Principal Ratings Health System Modernization Project I (P073974) Indicator ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Satisfactory Risk to development Negligible to Low Negligible to Low Negligible to Low outcome Bank performance Satisfactory Moderately satisfactory Satisfactory Borrower performance Satisfactory Satisfactory Satisfactory Note: The Implementation Completion and Results Report (ICR) is a self-evaluation by the responsible Global Practice. The ICR Review is an intermediate Independent Evaluation Group product that seeks to independently validate the findings of the ICR. PPAR = Project Performance Assessment Report. Health System Modernization Project II (P104467) Indicator ICR* ICR Review* PPAR Outcome Satisfactory Satisfactory Satisfactory Risk to development Negligible to Low Negligible to Low Moderate outcome Bank performance Satisfactory Satisfactory Satisfactory Borrower performance Satisfactory Satisfactory Satisfactory Key Staff Responsible Health System Modernization Project I (P073974) Management Appraisal Completion Project Team Leader Toomas Palu Susanna Hayrapetyan Sector Manager or Practice Manager Annin Fidle Abdo S. Yazbeck Sector Director or Senior Global Tamar Manuelyan Atinc — Practice Director Country Director Donna Dowsett-Coirolo Asad Alam Health System Modernization Project II (P104467) Management Appraisal Completion Project Team Leader Enis Barış Susanna Hayrapetyan Sector Manager or Practice Manager Armin H. Fidler Enis Bariş Sector Director or Senior Global Tamar Manuelyan Atinc Timothy Grant Evans Practice Director Country Director Donna Dowsett-Coirolo Mariam J. Sherman v IEG Mission: Improving World Bank Group development results through excellence in independent evaluation. About This Report The Independent Evaluation Group (IEG) assesses the programs and activities of the World Bank for two purposes: first, to ensure the integrity of the World Bank’s self-evaluation process and to verify that the World Bank’s work is producing the expected results, and second, to help develop improved directions, policies, and procedures through the dissemination of lessons drawn from experience. As part of this work, IEG annually assesses 20–25 percent of the World Bank’s lending operations through fieldwork. In selecting operations for assessment, preference is given to those that are innovative, large, or complex; those that are relevant to upcoming studies or country evaluations; those for which Executive Directors or World Bank management have requested assessments; and those that are likely to generate important lessons. To prepare a Project Performance Assessment Report (PPAR), IEG staff examine project files and other documents, visit the borrowing country to discuss the operation with the government, and other in-country stakeholders, interview World Bank staff and other donor agency staff both at headquarters and in local offices as appropriate, and apply other evaluative methods as needed. Each PPAR is subject to technical peer review, internal IEG panel review, and management approval. Once cleared internally, the PPAR is commented on by the responsible World Bank Country Management Unit. The PPAR is also sent to the borrower for review. IEG incorporates both World Bank and borrower comments as appropriate, and the borrowers ’ comments are attached to the document that is sent to the World Bank’s Board of Executive Directors. After an assessment report has been sent to the Board, it is disclosed to the public. About the IEG Rating System for Public Sector Evaluations IEG’s use of multiple evaluation methods offers both rigor and a necessary level of flexibility to adapt to lending instrument, project design, or sectoral approach. IEG evaluators all apply the same basic method to arrive at their project ratings. Following is the definition and rating scale used for each evaluation criterion (additional information is available on the IEG website: http://ieg.worldbankgroup.org). Outcome: The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently. The rating has three dimensions: relevance, efficacy, and efficiency. Relevance includes relevance of objectives and relevance of design. Relevance of objectives is the extent to which the project’s objectives are consistent with the country’s current development priorities and with current World Bank country and sectoral assistance strategies and corporate goals (expressed in Poverty Reduction Strategy Papers, Country Assistance Strategies, sector strategy papers, and operational policies). Relevance of design is the extent to which the project’s design is consistent with the stated objectives. Efficacy is the extent to which the project’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Efficiency is the extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared with alternatives. The efficiency dimension is not applied to development policy operations, which provide general budget support. Possible ratings for outcome: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, highly unsatisfactory. Risk to Development Outcome: The risk, at the time of evaluation, that development outcomes (or expected outcomes) will not be maintained (or realized). Possible ratings for risk to development outcome: high, significant, moderate, negligible to low, and not evaluable. Bank Performance: The extent to which services provided by the World Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of supported activities after loan or credit closing, toward the achievement of development outcomes). The rating has two dimensions: quality at entry and quality of supervision. Possible ratings for Bank performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. Borrower Performance: The extent to which the borrower (including the government and implementing agency or agencies) ensured quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development outcomes. The rating has two dimensions: government performance and implementing agency(ies) performance. Possible Ratings for borrower performance: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory, and highly unsatisfactory. vi Preface This is the multiproject Project Performance Assessment Report (PPAR) for the Adaptable Program Loan (APL) Health System Modernization series (comprising a first phase [P073974], a second phase [P104467], and an additional financing [P121728]). APL I was approved on June 10, 2004, supported by an International Development Association (IDA) credit of $19 million, a $1.25 million Policy and Human Resources Development grant from Japan, and counterpart funds from the government, hospitals, and communities for $2.0, $3.2, and $0.1 million respectively. By project closing on June 30, 2010, actual project costs totaled $29.4 million. APL II was approved on March 8, 2007, with estimated total project costs of $29.6 million to be financed by a $22 million IDA credit and $7.17 from the government, $0.3 million from the State Medical University (which was not disbursed), and $0.15 million from local communities. The initial financing of APL II was supplemented by additional financing through a $19 million loan from the International Bank for Reconstruction and Development, approved in December 2010, and $6.32 million of counterpart funds. The project closed on February 29, 2016, three years and two months later than originally scheduled, after being restructured three times to allow for completion of civil works. Actual disbursements were $22.71 million of the IDA credit and $18.98 million of the International Bank for Reconstruction and Development loan, with the difference due to exchange rate fluctuations. This PPAR serves the accountability and learning purposes of the Independent Evaluation Group (IEG). It evaluates the extent to which the APL series achieved its intended outcomes and offers an opportunity to draw lessons from the long-term engagement of the World Bank in reform of the Armenia health sector aiming to inform and guide future investments in the health sector. The APL series was selected for an in- depth field-based assessment due to its potential for learning from long-term engagement of the World Bank in health sector reforms; its clustering nature that allows coverage of multiple lending operation in the same country; and the relatively low coverage of previous IEG project evaluations in the country. This report was prepared by Mercedes Vellez, evaluation officer, with the support of short-term consultants. The findings of the report are based on a review of World Bank project documents (project appraisal documents, Implementation Completion Reports and IEG Implementation Completion and Results Report Reviews, World Bank Group country strategies, and relevant sector strategies) as well as other World Bank engagements in the health sector in Armenia; a review of external academic and policy literature; analyses of secondary data (demographic and health surveys and health vii management information systems); interviews with World Bank staff and relevant stakeholders; and site visits. A mission to Armenia was undertaken by Mercedes Vellez from September 24 to October 5, 2018, during which interviews were conducted with government officials and technical staff, health service providers, relevant development partners, and other involved persons. The mission included visits to the Ararat Medical Center, the rural ambulatory of Mkhchyan, and the health post of Mrgavet, which were chosen in consultation with the government and World Bank staff. IEG gratefully acknowledges all those who made time for interviews and provided documents and information and expresses its gratitude to the World Bank’s office in Yerevan for the logistical and administrative support provided to the mission. Following IEG standard procedures, a copy of the draft report was shared with relevant government officials and agencies for their review and feedback and no comments were received. viii Summary After it gained independence from the Soviet Union in 1991, Armenia saw many health indicators worsen. Public spending on health dropped sharply, and despite the high growth rates before the 2008 global economic crisis, average public spending on health, at 1.4 percent for 2000–07, was considerably lower than the average for middle-income countries (2.1 percent), Europe and Central Asian countries (3.4 percent), and European Union countries (6.1 percent). Utilization of health care services similarly trended downward, as out-of-pocket expenditures had to compensate for the decrease in public spending. Although in the early 1990s Armenia had outpatient visits per capita that were similar to those other European Union countries (about 6.8), use dropped to 1.8 visits in 2001. Similar trends were observed for inpatient care discharges. Moreover, access to health care services was unequal among socioeconomic groups as poor households tended to forgo health care because of affordability issues. Like other members of the Commonwealth of Independent States, Armenia inherited from the Soviet era an oversized health care system with widely distributed health facilities and abundant hospital beds. The health system was overly oriented toward curative and hospital care, which resulted in serious inefficiencies. During the mid- 1990s, the quality of services deteriorated, accompanied by severe shortages of drugs, medical supplies, and equipment. Facilities were not maintained and informal payments to medical staff were common due to the low wages of health personnel. By 2004, Armenia had 142 hospitals, including 44 in Yerevan. These hospitals drained scarce resources from a chronically underfunded system, while the quality of primary and secondary care continued to decrease due to the deteriorating infrastructure. Hospital use and efficiency remained extremely low. Primary health care (PHC) was delivered through a network of rural ambulatories and urban polyclinics, and provision of care was fragmented. Due to cultural beliefs and perceptions that PHC services were of poor quality, people tended to self-refer to hospitals and emergency care instead of seeking PHC services as a first contact. In the decade after independence, the government had identified major reforms of the health care system with the objective of improving efficiency. The health reform included a shift in financing of health care and reorienting the system toward PHC. Two important goals of the government plan were to strengthen PHC and consolidate hospitals to reduce inefficiencies and create health budget savings to be reallocated to improvements in the quality of health care. The World Bank supported the reform program through an adaptable program loan (APL) over 12 years, comprising two projects (the Health System Modernization Project, phases one and two) and additional financing in the second phase. The objective of the ix APL was to “improv[e] the organization of the health care system to provide more accessible, quality and sustainable health care services to the population, in particular to the most vulnerable groups, and to better manage public health threats” (World Bank 2004a, 5; World Bank 2007b, 14). Health System Modernization Project (APL I) As stated in the Development Credit Agreement of July 30, 2004, APL I objectives were “to support the Borrower’s first phase of the Program through expanding access to quality health care, improving the quality and effectiveness of selected hospital networks, and establishing capacity for health policy making and monitoring” (World Bank 2004b, 15). APL I development objectives are mostly consistent across project documents: improved access, quality, efficiency, and governance of health systems. The outcome of APL I is rated satisfactory. The relevance of objectives is rated substantial. APL I objectives were relevant for a health sector characterized by low use of health services, high out-of-pocket expenditures, poor quality of services, and system inefficiencies. The objectives were aligned with the priorities reflected in government strategies and legislation and in the World Bank’s Country Assistance Strategies. However, the objectives were not fully aligned with the APL program objectives: the focus on the poor—explicitly reflected in the APL program objectives and emphasized in government priorities and World Bank strategies—was absent from the project objectives. The relevance of design is rated substantial. APL I exploited experience from previous lending and knowledge instruments to support the health reform agenda. The APL instrument was appropriate as the government was committed to implementing a series of complex health reforms that required continuity. The reconstruction of the theory of change by the Independent Evaluation Group shows that project design followed a logical thread from activities to expected outcomes. Some determinants of health care use, however, were not fully addressed by project design. APL I, for instance, did not include behavioral change interventions to address health-seeking behaviors. The achievement of objective 1—expanding access to quality PHC—is rated substantial. APL I contributed to expanding the family medicine model for PHC. Increased structural quality resulted from enhanced qualifications for physicians and nurses through the provision of training, and infrastructure investments in selected PHC facilities. Other dimensions of quality were also enhanced, as evident by the increase in patient satisfaction with health care services and the improvement in detection rates for common childhood conditions. These investments in the PHC sector contributed to increased coverage of PHC services (total ambulatory visits per capita increased from 2.4 x in 2005 to 3.6 in 2010). Family medicine practice improved physicians’ capacities to handle certain specialized cases, but the actual role of family physicians as gatekeepers was less apparent in urban areas. The achievement of objective 2—improving quality and efficiency of selected hospital networks—is rated substantial. APL I helped the government to implement an ambitious hospital optimization plan and supported three hospital mergers in Yerevan and two in the marzes, which reduced excess capacity in the hospital network. In addition, the project support to hospital modernization through improvements in infrastructure and medical equipment helped raise structural quality. Beneficiary surveys showed that medical personnel and patients were highly satisfied with the facilities and improved quality of services. Hospital mergers also resulted in efficiency and productivity gains as evident in increased bed occupancy rates and reduced average length of stay. From the perspective of hospital balance sheets, however, it is not clear whether mergers resulted in net savings. The achievement of objective 3—laying groundwork for effective health sector policymaking and monitoring—is rated substantial. APL I contributed to strengthening institutional capacity for the evaluation of health sector performance through the development and institutionalization of two core instruments to inform decision-making processes: National Health Accounts (NHA) and Health System Performance Assessment (HSPA). NHA reports are essential to monitor health expenditure patterns. Before the World Bank’s support, systematic data on health expenditures by source were not collected. World Bank experts also contributed to developing capacities of the National Statistical Service in charge of producing high-quality HSPA reports. In public expenditure management, the project provided technical assistance for costing studies and reimbursement mechanisms and helped introduce performance-based contracts at the PHC level. Additionally, APL I was effective in supporting the State Health Agency to become a more advanced and capable purchasing agency as is clear from the increased pace of processing contracts with health providers. The introduction of new legislation on health waste management was also a contribution of APL I. Efficiency is rated substantial as cost-benefit analyses suggest that the project investments were good value for money based on a substantial net present value of $6.6 million and an economic rate of return of 11 percent at appraisal. Although overall project benefits outweighed the estimated costs, as evident in the positive rate of return, the family medicine model worked less well in urban areas mainly because of the availability of specialists in polyclinics and a preconception among patients that those specialists are more capable for treating certain conditions than generalists. During APL I, about 40 percent of retrained physicians (433 doctors) were from Yerevan, where the most physicians are, suggesting that the allocation of project resources could have xi been more selective based on the potential for doctors to effectively put into practice their new medical knowledge in the city. Risk to development outcome is rated negligible to low. Political risk was low because of government commitment and ownership of the health reform at the highest levels. The World Bank has built a strong and continued engagement in the health sector as evident from another lending instrument approved in that period and additional knowledge activities. The second phase of the program built on APL I and its objectives also reflected key performance dimensions of the health system. Bank performance is rated satisfactory. Quality at entry is rated satisfactory. The APL was an appropriate instrument, and the APL I design responded to the health sector needs of the country and was coherent with and benefited from other operations in the World Bank’s Armenia portfolio. In addition, the World Bank properly identified project risks and designed appropriate mitigation actions. Quality of supervision is also rated satisfactory. Missions were undertaken in coordination with other World Bank teams involved in budget support operations bringing coherence in the World Bank’s advice and dialogue with the country counterparts. The World Bank—and the Health Project Implementing Unit (HPIU)—systematically reported progress on output and intermediate outcome indicators, although reporting on health waste management by hospitals was limited. Borrower performance is rated satisfactory. The government was highly committed to health sector reform and gave high priority to the project as evident by the sustained health budget allocations (despite the effects of the global crisis on public finance), the timely allocation of project counterpart funds, and a series of policies that reinforced aspects of project design. Implementing agency performance is also rated satisfactory due to its highly qualified staff, who regularly monitored project performance and were perceived by stakeholders as performing financial and managerial work above the standards of the region. Health System Modernization Project (APL II) The second phase of the APL was approved two and a half years after the approval of APL I. As stated in the Financing Agreement of March 9, 2007, the objective of the project was “to strengthen the [Ministry of Health]’s capacity for more effective system governance, scaling up family medicine-based PHC and upgrading selected health care service delivery networks in the Selected Marzes to provide more accessible, quality and sustainable health care services to the population” (World Bank 2007b, 5). The Additional Financing of December 2010 did not change the core project objectives. APL II focused on key performance dimensions of the health care system: access, quality, sustainability, and governance. xii The outcome of APL II is rated satisfactory. The relevance of objectives is rated substantial considering they aligned with the needs of the health sector. The objectives were also relevant to country and World Bank strategies. Yet, as in APL I, ensuring equitable access to health services was not part of the APL II development objectives. The relevance of design is rated substantial. APL II core objectives and project components were kept from the previous phase, and the theory of change continued to be valid. APL II continued to support expansion of the family medicine model, infrastructure improvements of PHC facilities, and the implementation of the hospital optimization plan in the marzes not covered during the first phase. As in APL I, the project was complemented by other budget support operations. Momentum in hospital optimization reforms and government willingness to scale up its efforts led to the approval of APL II one year ahead of schedule. Most of the triggers were fully met, which was positive considering only two years had passed since project approval. However, the rapid transition from APL I to APL II may have limited opportunities for course corrections in the implementation of the family medicine model in urban areas. The achievement of objective 1—to provide more accessible health care services to the population—is rated substantial. APL II continued supporting improvements in access and structural quality of PHC services. The project helped the government improve access to and use of PHC services (per capita PHC visits increased from 2.4 in 2005 to 4.1 in 2017, and marzes supported by APL II registered a substantial increase in the number of per capita ambulatory visits). Concerns remained about the gatekeeper role of family physicians: in 2016, 36 percent of patients still went directly to specialists. Hospital optimization was carried out in one network per marz through upgrades to the physical infrastructure of medical centers. The increase in hospital use rates was countrywide but it was more pronounced in district hospitals in marzes than in Yerevan. The achievement of objective 2—to provide more quality health care services to the population—is rated substantial. APL II helped improve key health sector quality indicators. Beyond structural quality of PHC, process quality indicators also improved as evident by the increased share of patients having screening tests for selected noncommunicable diseases, as well as by the rise in detection rates for common childhood conditions during preventive care examinations. Improvements in health facility infrastructure and enhanced physician skills positively affected the perceived quality of care, albeit to different extents. For example, surveys conducted by the project in eight hospitals found high satisfaction rates among patients and physicians, while HSPA surveys found that positive perceptions of quality of care increased more in rural than urban areas for PHC services, and more in urban than rural areas for hospital services. xiii The achievement of objective 3—to provide more sustainable health care services—is rated substantial. The use of preventive and cost-effective health services increased, helping to contain the growing burden of noncommunicable diseases. The rationalization of hospital networks addressed system overcapacity (as evident in a substantial decrease in the number of beds in 14 hospitals supported by APL II, and a reduction by 85 percent in hospital area across all regions). Additionally, as in APL I, but at the regional level, hospital optimization led to efficiency and productivity gains in marz hospitals, as evident by a reduction in the average length of stay from 7.7 days in 2006 to 5.8 days in 2016 (higher than the decrease at the national level), and continuous progress in the average occupation per hospital bed. While there is no evidence on the extent to which net savings were achieved by mergers due to the lack of a detailed analysis at hospital level, it was estimated that maintenance costs resulting from new and modernized infrastructure would represent less than 2 percent of the public health budget. Additionally, public health spending as a percentage of gross domestic product (GDP) remained essentially constant during APL II at 1.6 percent despite the 2009 financial crisis. The achievement of objective 4—strengthen Ministry of Health capacity for more effective system governance—is rated modest. A culture of evidence-based impact assessment was established through the institutionalization of key documents to monitor health policy—the HSPA and NHA reports—and APL II contributed to introducing necessary adjustments to health financing mechanisms. However, the institutional development component lost traction because of a government decision to give priority to infrastructure investments over technical assistance and consultancy services. Efficiency is rated substantial. Unlike APL I, no cost-benefit analysis was done to assess the allocative efficiency of project investments. Yet, qualitative efficiency analysis offers positive results, including that APL II supported incentives for the use of preventive health care services to reduce the burden of relatively expensive care for late diagnosis of chronic diseases, which is an efficient strategy. Implementation efficiency was also strong. The limited application of the family medicine model in urban areas that continued during APL II was a shortcoming in the efficient use of project resources, but it accounted for only 6.5 percent of actual project costs. Risk to development outcome is rated moderate. External financing is critical for Armenia to sustain the achieved improvements in health service delivery and to ensure financial risk protection. The medium-term expenditure framework for 2019–21 projects increases in health sector financing over the next three years, but this financing continues to be projected at about 1.5 percent of GDP by 2021. The World Bank continues to support the health sector through a $35 million Disease Prevention and Control Project, which focuses on improving maternal and child health services, xiv strengthening prevention and management of selected noncommunicable diseases, and enhancing the efficiency and quality of selected hospitals. The upcoming Country Partnership Strategy envisages further health support. Bank performance is rated satisfactory. Quality at entry is rated satisfactory. The World Bank leveraged synergies with other budget support lending and worked in close coordination with other donors. The APL instrument continued to be adequate for implementing the needed reforms, and the World Bank team took advantage of its flexibility for accelerating the preparation of the second phase to seize positive political momentum. This acceleration, however, may have limited adaptive management to improve the performance of the family medicine model in urban settings. Quality of supervision is rated satisfactory. Strong engagement and fluid communications with government counterparts allowed the World Bank to react quickly on additional financing needs for more rehabilitation work in the marzes. Borrower performance is rated satisfactory. Government performance is rated satisfactory because of continued high commitment to the project as evident by the timely provision of counterpart funds even during the global financial crisis. Implementing agency performance is rated satisfactory. The financial, procurement, and supervisory functions of the HPIU were highly regarded by stakeholders. The HPIU continued to adequately monitor progress on results and implementation of project components. Lessons • An approach that exploits synergies and lessons from other World Bank engagements in the health sector is important for undertaking complex reforms and helping the government stay the course of the reform. Complementarities across lending and knowledge instruments allowed the World Bank to engage in a range of health policy areas, including health financing, governance and stewardship of health authorities, and service delivery. • Macro and micro health policies need to be combined in a manner that the unintended consequences of policy changes are not overlooked. For example, recurrent adaptations of the Basic Benefit Package—changes in services covered, entitled population groups, and the pricing system supported by policy-based lending—created uncertainty for patients about the boundaries of the benefit package, increasing the risk of informal payments and potentially undermining health care use. Similarly, the introduction of health financing policy changes, such as the global budgeting mechanism, while improving efficiency and cost containment from a macro perspective, may have had deleterious effects on some dimensions of quality (notably through the creation of waiting lists). xv • A shortened period between the approval dates of successive phases of an APL can limit the opportunity to incorporate lessons from previous phases into the design of new ones. The second phase of the operation was advanced to only two and a half years after the approval of the first phase. While this allowed the program team to seize the political momentum to implement the hospitals optimization plans in the marzes, it also limited the time to incorporate lessons from the first phase into the design of the second phase and introduce course corrections in the implementation of the family medicine model. By the end of APL I, it was clear that such a model was less suitable in urban areas because of the availability of specialists within the same facility. Yet, because of the limited time, the second phase did not include design components supporting private PHC practices in cities. • In country contexts with strong social and cultural factors affecting uptake of health care services, supply-side and systemwide policy reforms need to be combined with demand-side interventions addressing the health-seeking behavior of patients. Increased use of PHC services, especially by the vulnerable, depends on the extent to which services are accessible and affordable, have a minimum level of perceived quality, and on cultural factors that affect health- seeking behaviors. While the APL program addressed the accessibility and quality of health care services, attention to patient perceptions and preferences was not explicit, and a considerable share of patients continues to self-refer to specialists due to the preconception that those specialists are more capable of treating certain conditions than generalists are, illustrating that social and cultural preferences take longer to change. • While investments in infrastructure are not enough for health system modernization, they can help ensure acceptance of the proposed organizational changes involving strong stakeholders in the hospital sector. The enthusiasm of regional health authorities to pursue hospital mergers was strongly associated with the promise of major investments in the marz hospital networks. In fact, implementation of the program was particularly successful in those regions where it was followed up with major infrastructure investments in those networks. Auguste Tano Kouame Director Human Development and Economic Management Independent Evaluation Group xvi 1. Background and Context 1.1 Armenia experienced strong economic growth during the post-Soviet transition period. Armenia is a lower-middle-income country with a gross national income per capita of $3,770 (2016 data). The country has a population of 2.9 million people, 37 percent of whom lives in rural areas. The Armenian economy started to improve after the severe difficulties that followed its independence from the Soviet Union in 1991. After a series of economic reforms, the country grew at an average annual rate of 9 percent in the period 1998–2003. High growth, low inflation, and a stable currency led to a reduction of poverty rates from 55 percent in 1998 to 47 percent in 2001 (see appendix C, table C.1). 1.2 Following the post-Soviet transition period, public spending on health dropped sharply, as did health care use rates. Health care use rates declined in part because formal and informal out-of-pocket expenditures had to compensate for the decrease in public spending. Although outpatient visits in Armenia were similar to those of other European Union countries in the early 1990s (about 6.8 per capita), use dropped to 1.8 visits in 2001. Similar trends were observed for inpatient care discharges (see appendix C, figures C.1 and C.2). Moreover, 2001 survey results show disparities in health care use among urban and rural households (30.5 percent of sick or injured people sought care in urban areas compared with 26.1 percent in rural areas). The 2004 Integrated Living Conditions Survey confirms these inequalities by socioeconomic status: while 94 percent of the richest were treated when sick, only half of the poorest received treatment if needed; this is because low-income groups would forgo health care as they could not afford it (see appendix C, table C.4). Despite the economic improvements up to 2008 (when the global crisis hit economic growth), average public spending on health was only 1.4 percent of GDP in the period 2000–07, considerably lower than other middle-income countries (2.1 percent), Europe and Central Asian countries (3.4 percent), and European Union countries (6.1 percent; see appendix C, tables C.2 and C.3). 1.3 Like other members of the Commonwealth of Independent States, Armenia inherited an oversized health care system with widely distributed health facilities and abundant hospital beds. In the Soviet era, the organization, management, and finance of the health system was centrally coordinated through the Ministry of Health. Services were delivered through a territorially structured and hierarchical network of 182 hospitals (general and specialized) and 1,500 outpatient facilities (health posts, rural ambulatories, and polyclinics; World Bank 1997). The health system was overly oriented toward curative and hospital care, which resulted in serious inefficiencies. During the mid-1990s, the quality of services in Armenia deteriorated, accompanied by severe 1 shortages of drugs, medical supplies, and equipment. Facilities were not maintained and informal payments for health services were widespread due to the extremely low wages of health personnel. By 2004, Armenia had 142 hospitals, including 44 in Yerevan, which had a population of 1.2 million. The hospital network drained scarce resources from a chronically underfunded system, and the quality of primary and secondary health care continued to decline due to the deteriorating infrastructure. Hospital use and efficiency remained extremely low. 1.4 Poor quality of primary health care (PHC) reduced use of services. PHC was delivered through a network of rural ambulatories and urban polyclinics, and care was fragmented in different streams for adults (therapists), children (pediatricians), and women (gynecologists). Due to cultural beliefs and perceptions of poor quality of PHC services, people tended to self-refer to hospitals and emergency care instead of seeking PHC services as a first contact. In fact, use of PHC services had declined more than for hospital care, and outpatient contacts per person per year were among the lowest in the region. 1.5 Maternal and child health outcomes improved with economic growth but with persistent inequities and an increasing burden of noncommunicable diseases. Infant mortality rates fell from 26 per 1,000 live births in 2000 to 11 per 1,000 live births in 2017, and maternal mortality rates dropped from 40 per 100,000 live births to 25 per 100,000 live births in 2015 (see appendix C, table C.1). While these outcomes are comparable to other Commonwealth of Independent States countries with similar socioeconomic levels, they do not reveal inequities. Survey results suggest that the infant and under five mortality rates were about 1.5 times larger in rural areas than in urban ones. However, the increased burden of noncommunicable diseases imposed additional challenges on the already weakened health care system. By 2004, mortality rates of some diseases (for example, hypertension and ischemic heart disease) had been increasing for a decade, even though morbidity rates of these diseases had decreased. This was partially attributed to reduced access to health services and essential drugs (World Bank 2004a). 1.6 In the decade after independence, the government of Armenia had identified major reforms for the health care system with the objective of improving cost efficiency. The health reforms included a shift in health care financing, reorienting the system toward PHC. Two key goals were to strengthen PHC and consolidate hospitals to reduce inefficiencies and create health budget savings to be reallocated to improve the quality of health care. The Ministry of Health became a policymaking and supervisory body. The State Health Agency (SHA) was established in 1998 as a purchaser of publicly financed health care services. Armenia was the first country in the region to implement the single purchaser model, proposed at the time by the World Bank and World Health 2 Organization (WHO). Health care providers became managerially and financially autonomous and derived their income from annual contracts with SHA and private out- of-pocket payments (Richardson 2013). As the system decentralized and public service provision was reconfigured, operation and ownership of health services was devolved to provincial governments (hospitals) and local governments (PHC). In the reconfiguration, almost all pharmacies, dental services, medical equipment support, and several hospitals in Yerevan were privatized. 2. Health System Modernization Project (APL I) Relevance of the Objectives and Design 2.1 This project was the first of two planned operations, packaged as a seven-year adaptable program loan (APL), to support implementation of the health sector reform program. The overarching objective of the APL was to “improv[e] the organization of the health care system to provide more accessible, quality and sustainable health care services to the population, in particular to the most vulnerable groups, and to better manage public health threats” (World Bank 2004a, 5; World Bank 2007b, 14).1 Objectives 2.2 Project development objectives are broadly consistent across project documents. As stated in the Development Credit Agreement of July 30, 2004, the objectives were “to support the Borrower’s first phase of the Program through expanding access to quality health care, improving the quality and effectiveness of selected hospital networks, and establishing capacity for health policy making and monitoring” (World Bank 2004b, 15). The project appraisal document of May 13, 2004 states similar project objectives: “support the implementation of the … health reform program through (i) expanding access to quality primary health care; (ii) improving quality and efficiency of selected hospital networks; and (iii) laying groundwork for effective health sector policy making and monitoring” (World Bank 2004a). 2.3 Improved access, quality, efficiency, and governance are thus key performance dimensions of health systems reflected in the project development objectives. Although the Development Credit Agreement used the term “effectiveness,” the support to health system reform (involving strengthening of PHC and consolidating hospitals to reduce inefficiencies and create health budget savings to be reallocated to the improvement of quality of health care) indicates that “efficiency” as in the project appraisal document was the appropriate term. (See table B.2 for a comparison of program and project objectives across project documents.) 3 Relevance of the Objectives 2.4 The project objectives were relevant for the health sector needs. As illustrated in the background section, the post-Soviet transition was characterized by low use of health services, high out-of-pocket expenditures, poor quality of services, and system inefficiencies related to unnecessary referrals to outpatient specialist care for health conditions that could be treated through PHC as well as oversized and poorly maintained hospital infrastructure. 2.5 Project objectives were aligned with government priorities reflected in country strategies and legislation, as well as with the World Bank’s Country Assistance Strategies. APL I project objectives were grounded in the financing strategies for PHC and health developed by the government during the 1990s. Launched in 1996, the PHC strategy aimed at securing access to quality basic health services, particularly for the poor and those in rural areas. On health financing, the SHA was established in 1998 shifting the allocation of public funds from line item budget to contract-based payments for a defined package of basic health care benefits. A 2003 government decree established a hospital master plan for the city of Yerevan to pursue mergers in its hospital networks to provide both outpatient and inpatient specialist care, as well as host family doctor teams. The FY02–04 and FY05–08 Country Assistance Strategies and the 2003 Poverty Reduction Strategy Paper emphasized the need to improve and rebuild human capital, especially for the poor, for which improvements in the quality and access to health services were critical. 2.6 APL I objectives were not fully aligned with the overall APL program objectives. Enhancing capacity for effective policymaking and monitoring can be expected to result in better management of public health threats such as HIV/AIDS and noncommunicable diseases. However, the focus on the poor in the APL program objectives and in government priorities and World Bank strategies was not reflected in project-level objectives. This lack of emphasis on the most vulnerable population was also highlighted in a recent Independent Evaluation Group (IEG) health services evaluation: only 8 percent of health sector projects have equity-related objectives and, therefore, distributional impacts are rarely regularly monitored. The lack of focus on the poor in the objectives of APL I, despite its importance for the program overall, affects the relevance of objectives of APL I (World Bank 2018a). 2.7 The relevance of objectives is rated substantial. Design 2.8 To achieve its objectives, the project proposed a multilevel (national and regional), multisectoral (primary and secondary health care), and multi-intervention approach that focused on implementation of the family medicine model, the 4 optimization of the hospital networks, and strengthening government capacity for health sector policymaking and monitoring. Components 2.9 The project had four components. 2.10 Component A: Family Medicine Development (Appraisal: $7.1 million; Actual: $6.76 million). This component aimed to train well-qualified family doctors and family medicine nurses as the first-line PHC providers using internationally peer reviewed curricula; provide incremental support to train and retrain 980 family doctors and 980 family medicine nurses (estimated to meet 60 percent of the country’s needs); expand the PHC Development Program to improve PHC infrastructure beyond the 81 communities supported under the first health project, and further development of PHC guidelines relevant to family medicine, including primary and secondary prevention of avoidable mortality. The project aimed at financing renovation and equipment for training institutions; training for trainers; technical assistance for curriculum evaluation and improvement; tuition and stipends for the staff to be retrained as family doctors and family medicine nurses; development and publication of practice guidelines; medical equipment and supplies for PHC teams; rehabilitation of PHC infrastructure; and vehicles for PHC in remote communities. 2.11 Component B: Hospital Network Optimization and Modernization (Appraisal: $15.0 million; Actual: $19.5 million). This component was to support the development and implementation of hospital optimization plans in Yerevan. Investments sought to consolidate infrastructure and services; modernize management structures and improve management capacity; strengthen accountability arrangements; introduce quality assurance systems; and improve management of health care waste. Financing focused on relocating hospital services due to internal reorganization of networks; acquisition of medical equipment; technical assistance for managerial functions and quality assurance; training of management teams; information technology (IT) equipment for basic financial management systems; training and supplies for health waste management (HWM); and technical assistance to update regional health services masterplans. 2.12 Component C: Strengthening Government Capacity to Develop and Monitor Effective Health Sector Policies (Appraisal: $2.4 million; Actual: $2.0 million). This component aimed to strengthen the capacity of the Ministry of Health and its agencies to perform its major functions of policy development and implementation of monitoring, regulation, and oversight of the health sector. It supported the strengthening of governance and management structures of health facilities and the oversight function of regional government (marz) structures. The project aimed at building capacity for evaluation of health sector performance by developing core monitoring instruments that 5 were needed to inform decision makers (Health Sector Performance Assessment [HSPA] and National Health Accounts [NHA]); improving public expenditure management in the health sector; strengthening the legal and regulatory environment, improving quality assurance mechanisms; raising public awareness about health reforms; and improved surveillance of HIV/AIDS and other public health threats complementing grant funds from the Global Fund to Fight AIDS, TB, and Malaria. 2.13 Component D: Project Management (Appraisal: $1.0 million; Actual: $1.1 million). This component supported establishment of the Health Project Implementing Unit (HPIU) to pursue strategic planning, operational management, and monitoring of project activities within the Ministry of Health. Financing was focused on key staff of the project unit; acquisition and rehabilitation of a project unit office; office equipment and supplies; and incremental operation costs. Relevance of Design 2.14 The project exploited experience gained from previous lending and knowledge instruments to support Armenia’s health development agenda. The health sector reform had been supported by the World Bank from its outset in the mid-1990s. Organisation for Economic Co-operation and Development data shows the World Bank was the biggest single donor in health during 2000–16 committing $90 million, 44 percent of commitments from all donors and 71 percent among multilaterals (see appendix C, table C.5). Since 1996 the World Bank provided budget support lending through five Structural Adjustment Credits, the first investment project in health in 1998, and several pieces of relevant economic and sector work. (See appendix C, table C.6, on the health- related portfolio of lending and nonlending projects in Armenia 1996–2017.) 2.15 The Structural Adjustment Credits addressed one of the main constraints for health system performance in Armenia: chronic underfunding of the public health system. Budget support operations aimed at progressively increasing the share of public spending allocated to health, especially in PHC, while ensuring financial sustainability and more efficient allocation of public resources. The need for a decline in hospital capacity, which significantly exceeded both the demand and budget, was imperative. The Structural Adjustment Credits also supported the strengthening of financial planning and budgeting capacities of the SHA in managing a contracting mechanism to ensure sustainability and no accumulation of arrears in payments for health providers in a capped global budget setup. Improving health system governance and budget management comprised the design (and subsequent revisions) of a basic benefit package (BBP), along with payment mechanisms for the delivery of health care services. 2.16 The BBP’s list of services covered, the population entitled to receive free of charge services, and the pricing system to control the global health budget were updated 6 annually. Such recurrent adaptations had unintended consequences since the lack of clarity on the boundaries of the state-funded package of benefits created uncertainty for people about which services were covered. This also increased the risk of informal payments, because it created an opportunity to levy charges for services that should have been covered (Rechel, Richardson, and McKee 2014). As use of health services depends on affordability issues, uncertainty increases the risk of increasing out-of- pocket payments and thus deters health care use. 2.17 The project entailed a great deal of continuity with the first Health Financing and PHC Development investment project approved in 1998. This project supported the incipient implementation of the government’s recent PHC and health financing strategies through the strengthening of PHC (starting the family medicine program, and support for guidelines, training, and physical capacity); and the introduction of output- based payments for health providers and the development of the BBP. The introduction of family medicine as the first point of contact was key to PHC reform (Hakobyan et al. 2006). 2.18 IEG’s reconstruction of the theory of change shows that the project design followed a logical thread between activities and expected outcomes (figure 2.1). To achieve the objective of expanding access to quality PHC services the project supported expansion of the family medicine model at national level, which implied expansion of the scope and context of PHC services. Health personnel were retrained, and the working environment was improved through rehabilitation of infrastructure and provision of basic equipment. These structural quality investments were expected to enhance the gatekeeper role of PHC physicians, reduce more expensive specialist and hospital referrals, and expand doctors’ management of chronic conditions. 2.19 To improve quality and efficiency in hospital networks the project supported the government’s hospital optimization and modernization plan for Yerevan. The principles guiding the selection of hospital mergers involved political will, proximity of structures, potential for cost reductions and use increases, among others. The civil works and facilities renovations, along with the organizational changes brought about by the optimization, were expected to increase structural quality and reduce duplication and management costs. The outright closing of some entire facilities would bring costs savings, reduce available beds, and result in staff redundancies. The project also envisaged implementation arrangements for staff reductions (natural attrition, incentive package for early retirement, relocations, and mobility grants) with support from the National Employment Service Agency. While not explicitly highlighted as an unintended outcome of the project, hospital mergers in Yerevan would potentially have a demonstration effect in other regions where hospital optimization and modernization were also needed. This was indeed supported later by APL II. 7 2.20 To improve governance for health sector policy making and monitoring APL I supported systemwide interventions. These included building capacity of Ministry of Health and public agencies to conduct diagnostic analysis and health system performance evaluation; provide technical assistance to improve management of public health expenditures and annual budget allocations; and support the SHA purchasing role and contracting mechanism with health providers to improve accountability and transparency of public spending. 2.21 The link between supply-side interventions and increased use of PHC services, especially by the vulnerable, depends on the extent to which services are affordable, have a minimum level of perceived quality, as well as on cultural factors that affect health-seeking behaviors. Beyond supply-side improvements aiming to increase patients’ perceptions of quality, other determinants of health care use were not fully addressed by the project design. Affordability issues (that is, health financing of a basic package of health services provide free to the poor and vulnerable) were supported by the World Bank through the Structural Adjustment Credits, as well as other donors. However, the project did not include behavioral change interventions to address health- seeking behaviors. 2.22 Another important assumption in the theory of change is the existence of a conducive organizational structure for the newly trained family physicians to apply their new knowledge. In rural areas, PHC physicians operate in rural ambulatories, small facilities that offer a reduced number of services. Health service delivery in urban areas is different. There, PHC services are delivered in polyclinics with a supply of specialists in the same facility or a nearby hospital. The proximity of specialists can create tensions and overlapping roles among physicians. The administrative mergers between polyclinics, hospitals, and maternities may have reinforced this tendency and thereby interfered with the ability of family doctors to practice their discipline. Despite this, project design overall took a holistic perspective that exploited the synergies and lessons from other World Bank engagements in the health sector. 8 Figure 2.1. Health System Modernization: Theory of Change Source Independent Evaluation Group. Note: Results Framework indicators of APL II are located along the diagram and are almost identical to those of APL I (see appendix C, table C.7 for list of indicators for APL I and II). APL = adaptable program loan; BBP = basic benefits package; HSPA = Health Sector Performance Assessments; M&E = monitoring and evaluation; MOH = Ministry of Health; NCD = noncommunicable disease; NHA = National Health Account; SHA = State Health Agency. 9 2.23 The project activities complemented those of other development partners. United States Agency for International Development (USAID) projects focused on enhancing the capacities of rural health posts (small health facilities staffed with nurses), whereas the World Bank supported rural ambulatories (health facilities staffed with PHC physicians and nurses). USAID also supported health financing efforts by providing technical assistance to the SHA to strengthen contracting mechanisms for purchasing BBP, including a performance-based contracting scheme for PHC providers to be introduced in the future. The World Bank and WHO worked closely on the provision of technical assistance for the development of NHA and the HSPA. The project envisaged the provision of complementary technical assistance to improve the capacities of the public health surveillance system also supported by grant funds from the Global Fund to Fight AIDS, tuberculosis, and malaria. 2.24 The APL instrument was appropriate as the government was committed to implement a series of health reforms that were complex in scope and that required accompanying resources and institutional support in a gradual and consistent manner. The government’s goal to expand access and improve the quality of health care services would take a long time, requiring consistency. The scope of reforms was broad as it included changes in health financing, provider payment systems, organizational changes in service provision, and balance among the different levels of the system. Moreover, the hospital optimization plans required transparency, ample consultations, mitigation of social consequences and, most importantly, time to build consensus and convince stakeholders of the reform benefits. The APL was an adequate choice due to its phased approach that would allow for sequential deepening of reforms over an extended period. It would also provide incremental resources to accelerate the implementation of key policy and institutional reforms allowing the World Bank continued leverage for contributing to solve difficult structural issues. 2.25 Triggers for moving to phase two of the program were adequately defined. As shown in see appendix C, table C.8, APL I established five triggers, subdivided into 13 indicators, that were well-defined, quantifiable, and easily measured. Consistent with good practice in APLs (see appendix C, box C.1), the triggers mirrored project design as they assessed progress in key intervention areas: hospital optimization, family medicine model, and institutional capacity strengthening. Moreover, the choice of triggers was balanced because it reflects good progress in inputs, institutional development, and outcome attainment.2 Triggers were also consistent with the project’s monitoring and evaluation design: seven of the 10 project outcome indicators are triggers, and trigger targets are consistent with the planned evolution of project outcomes. 2.26 The relevance of design is rated substantial. 10 Implementation Key Dates 2.27 The project was approved on June 20, 2004, became effective on December 14, 2004, and closed on June 30, 2010. The midterm review took place on November 27, 2006. The closing dates of the International Development Association (IDA) Credit and a Japan Policy and Human Resources Development Fund (PHRD) grant were extended once for 12 months for the credit (letter of February 24, 2009) and 16 months for the grant up to June 30, 2010. The credit was extended to complete civil works for two regional hospitals (Harazdan Medical Center and Ijevan Medical Center) and to disseminate the 2009 health reports. The PHRD grant was extended to allow for completion of technical assistance contracts. Planned versus Actual Expenditure by Component 2.28 The total cost estimated at appraisal was about $25.5 million, which was expected to be financed by the World Bank with a $19 million IDA credit, the government of Japan through a $1.25 million PHRD grant, and counterpart funds from the government, hospitals, and communities for $2.0, $3.2, and $0.1 million respectively. Actual project costs were $29.4 million with the increased cost mostly financed by greater hospital contributions ($2.5 million more than originally envisaged) and IDA credit increases due to exchange rate variations. Meanwhile, the government contributed 92 percent of originally committed funds. 2.29 Most of the increased funds were allocated to component B (Hospital Network Optimization and Modernization) that received additional financing of $5.4 million. Even though one of the four Yerevan hospital mergers included in the original design did not take place, the actual cost of the relevant component was still higher than appraised due to the higher-than-expected cost of civil works in the hospitals, and the addition of two marz hospital mergers. The Japan PHRD grant for institutional capacity building was used to finance several activities under component C (Strengthening Government Capacity; table A.1). Implementation Experience 2.30 Project performance ratings according to the Implementation Status Reports remained satisfactory throughout the project life. Project execution was successful, and the disbursements profile closely followed the original design. The central and local governments were highly committed to the needed reforms. Project implementation was well coordinated by the HPIU team, most core members of which had been with the project from the outset. The World Bank also regularly monitored project implementation and progress toward outcomes. Acceleration in the preparation of APL 11 II is a reflection that the project triggers for moving to the second phase were mostly met. Safeguards Compliance 2.31 Project design adequately incorporated safeguard measures to manage potential environmental impacts. The project was classified as Category B, the safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered, and an Environmental Management Plan (EMP) was prepared in consultation with key stakeholders to mitigate potential risks associated with the project’s construction activities. 2.32 The project followed good practices in environmental risk prevention. A World Bank tool kit on mainstreaming environmental management (World Bank 2012) suggests that when the borrower has inadequate legal or technical capacity to carry out key functions (such as environmental monitoring, inspections, or management of mitigatory measures) the project should include components to strengthen that capacity. Indeed, capacity building activities to enhance environmental expertise within the HPIU, and HWM within hospitals, were included in project components D and B, respectively. Site- specific environmental screenings were part of the project’s implementation and supervision processes. 2.33 Implementation of the EMP was mostly satisfactory. Considering that standards in HWM were nonexistent in Armenia, the passing of national regulation was an important achievement of the project with effects beyond project-supported hospitals. Provision of training, equipment, and supplies to hospitals was key in building their capacity and improving waste segregation practices (see appendix C, box C.2 for details). However, there were some concerns regarding homogenous HWM practices across the country. Some interviewees highlighted the inadequate handling of solid waste, where material that could be incinerated either ended up in local landfills or was burned directly on- site, and inadequate handling of liquid waste, as wastewater was only minimally treated. Despite this, interviewees pointed out there were no outbreaks of infectious diseases from hospital sources, and hence no evidence of negative environmental impacts from the project’s activities. In terms of supervision, state inspection bodies were responsible for monitoring compliance of hospitals’ operation with the national standards, while the HPIU was expected to prepare quarterly reports, describing progress in the implementation of the EMP and reporting environmental issues arising from project activities. Currently, the HPIU does not have an environmental specialist devoted to this task. Financial Management and Procurement 2.34 No financial management issues occurred during project implementation. Project documents reported the HPIU conducted adequate accounting, reporting, budgeting 12 and planning, internal controls, funds flow, and staffing arrangements. The institutions providing family medicine training (Yerevan State Medical University, National Institute of Health, and the Yerevan State Basic Medical College) also had appropriate payment and recordkeeping systems to facilitate necessary accounting functions. Counterpart funds from the government were contributed as planned, and project disbursements closely followed the planned schedule. The ICR states “compliance with fiduciary requirements” (World Bank 2010, 26). 2.35 Procurement was managed effectively. The procurement unit of the HPIU was staffed with three specialists who managed acquisitions related to civil works, biomedical equipment and supplies, office equipment and furniture, and consulting services. The World Bank’s regional procurement adviser granted a special waiver to allow direct contracting with the three state-owned training institutions mentioned earlier in this section. Achievement of the Objectives Objective 1: Expanding Access to Quality Primary Health Care Outputs 2.36 The project contributed to expanding the family medicine model for PHC. As a first step toward increasing knowledge and capacity of physicians in the family medicine discipline, the project strengthened the capacity of the training institutions responsible for delivering the courses. Training centers in Shirak, Kori, and Syunik marzes (regions) and Yerevan were renovated and received medical equipment, furniture, IT or office equipment, skill labs, and learning materials. Training of trainers was provided to 60 family physicians and 30 nurses. Guidelines developed by the National Institute of Health and the State Medical University were printed and distributed across health facilities in Yerevan and marzes. 2.37 The qualifications of physicians and nurses were improved through training, contributing to enhance PHC quality. According to project data, 1,082 doctors and 988 nurses completed training in family medicine by 2010, exceeding targets. An external midterm evaluation concluded that the training program was an excellent example of a successful family medicine model, as all the main elements and principles of modern education were in place (adequate curriculum, teaching modality, and so on). The one- year duration of the program, longer than comparable programs in other Eastern European countries, provided doctors with new skills. The holistic approach to health care system reform, which combined doctor training with other structural changes, was highlighted as a factor of success as doctors could apply their new knowledge in 13 practice, where they now had adequate premises, equipment, and organizational support (Švab 2006). 2.38 PHC infrastructure for selected facilities was much improved and contributed to increased structural quality. While further support to infrastructure was envisaged in APL II, 6 rural ambulatories were constructed and another 14 rehabilitated and equipped with furniture and medical devices. Twenty PHC facilities in remote areas also received vehicles. Because PHC facilities were relatively neglected during the Soviet period, the World Bank’s financial assistance to rehabilitate and build facilities has significantly improved the situation (Richardson 2013). However, advances in licensing and accreditation were left for the next phase. Provisions for quality assurance of health care facilities and professionals were envisioned in the Health Care Law and the Ministry of Health planned to create a separate agency for licensing. Outcomes 2.39 Project investments in the PHC sector contributed to increase coverage of PHC services. Project data show that the share of the population covered by qualified family medicine physicians increased from 17 percent in 2004 to 85 percent by the end of the project, reaching the entire Armenian population as of September 2018. Data collected during the IEG mission show a positive evolution in the number of ambulatory visits to PHC. Total visits (excluding obstetric care) increased from 7,731 in 2005 to 11,595 in 2010, an increase from 2.4 to 3.6 visits per capita, respectively (see appendix C, table C.9). 2.40 Family medicine practice improved the capacities of physicians to handle specialized cases, but the role of family physicians as gatekeepers was less apparent in Yerevan. Qualitative data collected from stakeholders’ interviews widely confirmed that the family medicine model worked less well in Yerevan. The referral rate to specialists in project areas decreased from 32 percent in 2004 to 25 percent by the end of the project, according to project documents. This is a positive result, but these estimates do not distinguish Yerevan from other regions where the supply of specialists is more limited. According to the 2009 HSPA survey, a considerable share of the population still bypasses family doctors and go to hospitals for nonemergency conditions: 43 percent of referrals to hospitals were by individuals, 39 percent were by specialists, and only 20 percent were by family doctors and district physicians (Armenia, NSS and MOH, and ICF International 2012). 2.41 The lack of a conducive working environment for family physicians, physicians’ reluctance to abandon previous practices, and patients’ cultural beliefs explain the challenges facing family medicine model in the city. PHC services in Yerevan are delivered in polyclinics with a supply of specialists in the same facility or a nearby 14 hospital. The proximity of specialists creates tensions and overlapping roles among physicians. The administrative mergers among polyclinics, hospitals, and maternities may have interfered with the ability of family physicians to practice their new discipline. Some attempts have been made to establish independent family medicine practices, but economic barriers (including rental costs and owning specialized equipment largely available at polyclinics) have limited such experiences in the city. Moreover, the PHC strategy sought to integrate various streams of PHC (that is, children, adult, women) into the institution of family doctor. But traditional roles persist in Yerevan and family physicians still concentrate either on children or adult care according to whether they were pediatrician or therapist practitioners before (Richardson 2013). Regarding cultural factors, stakeholders largely agreed on the society’s preconception that narrow specialists are more capable for treating certain conditions and thus patients ask for referrals or seek a second opinion. 2.42 In addition to improvements in structural quality of PHC services through project infrastructure investments, other dimensions of quality of health care also improved. Regarding patients’ perceptions, the percentage of the population in project areas rating quality and access to PHC services as satisfactory increased from 87 percent to 95 percent according to project data. Regarding process quality, secondary data collected during the mission shows that detection rates for common childhood conditions during preventive care examinations improved, suggesting a higher quality of health care (see appendix C, table C.10). 2.43 Achievement of objective 1 is rated substantial. Objective 2: Improving Quality and Efficiency of Selected Hospital Networks Outputs 2.44 The project helped the government in implementing an ambitious hospital optimization plan. Three hospital mergers took place in Yerevan and two in the marzes.3, 4 As a result of the mergers, hospital space was reduced by 19,181 square meters. Ratios of hospital beds to population that were significantly higher than in industrialized countries were reduced nationwide (see appendix C, table C.11). The integration of administrative structures and the reorganization of clinical departments contributed to the elimination of duplication and overlap in administration and maintenance, diagnostic capacity, and clinical departments. 2.45 Hospitals’ modernization contributed to improved structural quality. As part of the hospital mergers, the project financed necessary civil works, the provision of modern medical equipment, including medical waste equipment, and the establishment of management information systems and provision of IT equipment. 15 2.46 APL I also supported the introduction of hospital management system tools. About 726 hospital staff at all levels (83 percent of hospital management) received training on management (that is, assessment of capacities, use, and patient flows), information systems, and HWM. A quality assurance system was designed and institutionalized, including satisfaction surveys for patients and health personnel and the establishment of supervisory committees in all marz hospitals. Outcomes 2.47 Efficiency and productivity gains were achieved through hospital mergers. According to project data, bed occupancy rates increased in all project hospitals ranging from 22 percent in 2004 to up to 88 percent in 2009. Productivity measures indicate that the ratio of full-time equivalent staff per 1,000 patient days decreased in all project hospitals, although its magnitude varied across mergers (see appendix C, table C.12). The three mergers in Yerevan represented about 25 percent of total bed capacity in the city. Additional data collected during the IEG mission show progress in the average occupation per hospital bed increasing from 169 to 225 days per year during APL I (see appendix C, table C.11). This is a result of a combination of both a reduction in beds and more use of hospital services. Figure C.3 in appendix C shows the negative trend over time of bed capacity along with the positive trend in number of discharges. 2.48 Efficient use of hospital beds and higher quality of hospital services led to a decrease in the average length of stay (ALOS). The reduction in ALOS was greater in project hospitals (from 9.5 days to 7.3 days) than in the country overall (from 10.27 days to 8.6 days) in 2004–09 (see appendix C, table C.12 and figure C.4). Although ALOS has been declining globally, in Armenia it dropped more sharply than in other European Union countries and Commonwealth of Independent States (see appendix C, figure C.5). In addition to improvements in hospital capacities, surveys revealed that medical personnel and patients were highly satisfied (about 85 percent) with the facilities and improved services pointing to quality improvements that may also have contributed to a reduction in ALOS. 2.49 Reductions in excess hospital capacity and in duplication of management costs after mergers may not necessarily translate into net savings in hospitals’ balance sheets due to enhanced costs associated with higher quality of care. APL I included a diverse set of compensation strategies for envisaged staff reductions due to mergers, but in practice, few staff dismissals took place, and those were mainly associated with retirement and contract termination (World Bank 2010). Thus, the hospitals’ mergers accommodated previous staff even at the expense of creating artificial positions, such as former directors becoming deputy directors. Nonetheless, new and renovated buildings, and modernized equipment, required additional expenses for maintenance, which would affect the cost of providing hospital services. Although NHA data show a slight 16 upward trend in the share of public health spending allocated to hospitals (see appendix C, figure C.6), the distribution of the health budget among hospitals has not necessarily followed the increased resource needs, according to key informants.5 2.50 Progress in hospital mergers in the city of Yerevan had a positive demonstration effect in other marzes where optimization was also a priority. Indeed, the government issued a 2006 master plan for hospital optimization in the remaining regions, which accelerated preparation of the next APL II to seize the political momentum. The enthusiasm of regional health authorities to pursue mergers was strongly associated with the promise of major investments in marz hospital networks. 2.51 Achievement of objective 2 is rated substantial. Objective 3: Laying Groundwork for Effective Health Sector Policy Making and Monitoring (Governance) Outputs 2.52 The project strengthened of the health sector governance, decision-making, and monitoring in at least four areas: evaluation of health sector performance, regulation and legislation, hospital management, and public expenditure management. 2.53 APL I contributed to strengthening institutional capacity for the evaluation of health sector performance. Technical assistance supported the development of two core instruments to inform decision-making processes, the NHA and HSPA. NHA are essential to monitor health expenditure patterns. Before the World Bank’s support systematic data on health expenditures by source were not collected. The World Bank funds supported technical assistance for the development of questionnaires for household surveys, for which public resources would not be sufficiently available. The World Bank, WHO, and USAID helped create a working group for NHA with high- quality experts at that time. Built capacities remain today, as some members of the former working group continue working on NHA and the Armenian National Institute of Health keeps developing its capacity and has the necessary staff. NHA reports have been produced and published annually since 2004. World Bank experts also helped develop the capacity of the National Statistical Service in charge of producing the HSPA. The HSPA reports have been developed, published, and distributed every two years. The World Bank team judged that the quality and frequency of these reports are better than in many other comparable countries. Moreover, this Project Performance Assessment Report (PPAR) presents data based on these reports. 2.54 A remarkable achievement was the development of regulations on HWM for health facilities. Before the project there was no such legislation. The project helped develop guidelines and procedures that materialized in the Ministry of Health Decree 17 N03-N on HWM in 2008. Hospital staff were also trained in the application of the new HWM rules. Based on the new legislation, health facilities contracted with licensed waste treatment companies, and created a new job position to take responsibility in this area, ideally to be filled by an epidemiologist. 2.55 Along with optimization and modernization, hospital management was strengthened. The project helped develop the curricula for management and health care governance courses for different management levels, including the provision of training of trainers at the National Institute of Health management faculty. About 730 health managers were trained in management, accounting, and financial systems. Financial management and accounting procedures were updated in all hospitals, and necessary IT equipment was also provided. Although Hospital Supervisory Committees were established in all marz hospitals, as described in government decree N-1187-N of May 19, 2005, their effective functioning was not sustained because the remuneration of members depended on already insufficient hospital budgets. Similar budgetary reasons limited the application of external independent audits of public hospitals. Only three hospitals in Yerevan participated in such external audits because their annual revenues exceeded Armenian dram 1 million, which was threshold for making audits legally mandatory. 2.56 In public expenditure management, the project has been effective in supporting the SHA to become a more advanced and capable purchasing agency. Technical assistance assessing the organizational and governance arrangements of the SHA concluded that the agency should have a more strategic purchaser role, rather than functioning as a mere contracting and payment agency. Initially the SHA was an independent institution, however in 2002 it was included in the structure of the Ministry of Health. As part of these efforts the project commissioned a study by international experts to calculate the real cost of delivering the BBP. The project helped introduce incentive contracts to improve performance at PHC level and promote the provision of preventive services by family physicians. In collaboration with USAID, performance- based indicators were selected to be implemented with the 2011 contracts. Also, the MIDAS software used by SHA for reporting of services provided (originally developed with support from USAID) was further upgraded and expanded into a MIDAS-2. Outcomes 2.57 There is not a clear quantitative outcome indicator to reflect the effectiveness of institutional strengthening interventions. The PPAR team made efforts to find new ways of bringing up evidence on the impact of strengthening the country’s capacities through technical assistance beyond the project results framework. 18 2.58 In line with the described outputs, stakeholders consistently said that project investments contributed to substantial institutional development impacts at the different levels of the health system. The Ministry of Health and other state agencies (the SHA, National Statistical Service, National Institute of Health, former National Center for Disease Control and Prevention, health facilities) benefited from capacity building activities as each of the project components supported institutional strengthening in their respective areas. The purchasing agency increased the pace in processing contracts with health providers thanks to the use of software. All SHA contracts were signed in a timely manner (that is, before February each year, about 30 days after the global health budget was approved). Also, the total number of contracts signed decreased from 125 in 2004 to 106 in 2009 as a result of hospital mergers. 2.59 The institutionalization of the NHA was an important achievement. The World Bank team assessed that the quality content, as well as the frequency of publication of these reports, are better than in many comparable countries. NHA is published at the end of each year in both Armenian and English. The reports are being disseminated among the stakeholders. Most interviewers agreed that these reports are used for decision-making. For example, the Ministry of Health financial department and Ministry of Finance are using the NHA for the state budget development. 2.60 The introduction of new legislation on HWM was an important contribution of the project. There were no legal acts that directly regulated the field of medical waste in Armenia. The introduction of mandatory contracting of licensed waste collection enterprises generated additional demand that likely had spillover effects in the development of the market for such services. Until 2008 there were no licensed companies for health waste treatment. 2.61 There is a consensus among interviewed stakeholders that monitoring and reporting became more widespread and systematic. This includes financial and accountability systems in hospitals, costing studies and reimbursement mechanisms, and so forth. Yet the use of all these data and studies varies across health agencies and tools. Some interviewees suggested that some decisions are still made subjectively based on populistic approaches and political interests. 2.62 Achievement of objective 3 is rated substantial. Efficiency 2.63 Cost-benefit analyses (CBA) suggest that the project’s investments provided good value for money overall. The CBA estimated a net present value of $6.6 million and an economic rate of return of 11 percent at appraisal. The analysis focused on the benefits and costs of investing resources in the PHC component and the hospital 19 modernizations in Yerevan, which represented 87 percent of the total project costs. Benefits derived from hospital mergers included savings from staff reductions, savings in space costs (utilities), savings in rent for vacated property, decrease in unnecessary hospital stays, and averted productivity losses due to ALOS reduction. Benefits related to the PHC component included reduction in unnecessary hospital admissions, averted productivity losses due to unnecessary hospital admissions, reduction in referral rates to outpatient specialist care, averted productivity losses due to less referrals for rural population, reduction in travel costs for rural population, and potential life years saved due to reduced mortality from noncommunicable diseases and respiratory conditions. 2.64 As a good practice, the CBA estimated the economic returns of each component, showing that investments in PHC yielded higher returns (net present value $14.7 million and economic rate of return 47 percent over a 10-year period) than the hospital mergers (Economic rate of return −3 percent over 10 years and 8 percent over 20 years). This is not surprising since hospital renovations involved large upfront costs that require a longer time to be offset by the benefits. As was highlighted in the CBA, monetary benefits from staff reductions would be small because of the low salaries. These reductions also may have been overestimated because, in practice, most staff were kept. 2.65 The ex post CBA dropped some benefits included at appraisal (including reduced staff costs and averted productivity losses) due to lack of available data. However, it included SHA’s efficiency gains in purchasing health care services due to IT investments. No details are provided on the assumptions of such calculations. The net present value was $7.6 million ($20.7 million) for a 10-year period (20-year period) and a discount rate of 5 percent (10 percent). The fiscal burden of project investments was considered negligible as the public health spending remained at 1.7 percent of GDP in 2010, as at the beginning of the project. Yet CBA seems to include operation and maintenance costs for equipment purchased and facilities newly constructed under the project. 2.66 As noted in the previous section, the family medicine model worked less well in Yerevan mainly for lack of a conducive environment. Still, 40 percent of retrained physicians during 2005–09 (representing 433 doctors) were from Yerevan according to project documents. The allocation of project resources in this area could have been more selective based on the potential for doctors to effectively put in practice the acquired new medical knowledge. 2.67 The overall efficiency rating is substantial. 20 Outcome 2.68 Overall outcome rating is satisfactory. The relevance of objectives is rated substantial because they were highly aligned with country needs and priorities, though not reflecting an explicit focus on the poor. Relevance of design is also substantial on the basis of a sound theory of change and a holistic perspective that built on lessons from previous engagements in the health sector. The achievement of the three objectives is considered substantial due to the considerable contributions of the project to increasing coverage and quality of PHC and hospital services; to improving hospital efficiency; and to strengthening governance in evaluation of health sector performance, regulation and legislation, and hospital and public expenditure management. Efficiency is also rated substantial as the project investments demonstrated good value. Risk to Development Outcome 2.69 Risk to development outcome is rated low. Political risk was low because government commitment and ownership of the health reform were high. The World Bank has built a strong and continued engagement in the health sector as evident by the health portfolio described in the project design section (see also appendix C, table C.6). The second phase of the program built on APL I and its objectives also reflected key performance dimension of the health system (access, quality, efficiency, governance). Bank Performance Quality at Entry 2.70 The World Bank’s performance on quality at entry was satisfactory. Project design responded to the country health sector needs and was aligned with government priorities regarding the increase of access, quality, and efficiency of health care services systemwide. As Armenia continued expanding the family medicine model, optimizing the hospital network, and building institutional support, project design was built on lessons learned from similar experiences in the region and a previous World Bank health operation in Armenia. Moreover, the project was coherent with and benefited from other operations in the World Bank’s Armenia portfolio. Prior actions of previous and contemporary budget support operations reinforced various aspects of the project- supported health reforms. 2.71 The APL instrument was an appropriate choice for this project considering the broad range of reforms the government was committed to implement, and the need for phased influx of resources and accompanying institutional support. In addition, the World Bank properly identified project risks and designed appropriate mitigating actions, including the development of the EMP according to environmental safeguards. Moreover, the design incorporated actions related to the HWM, including support in 21 developing and passing national regulations in this area, and setting up management structures, expertise, and equipment in project hospitals to implement those regulations. Quality of Supervision 2.72 Bank performance on quality of supervision was satisfactory. Supervision missions were conducted about twice a year in conjunction with design or supervision missions for the Poverty Reduction Support Credit series or the Structural Adjustment Credit operations. This internal coordination was not only efficient and less onerous for the country client but also afforded coherence in World Bank advice and dialogue with counterparts, including the Ministry of Finance and Ministry of Health. Task leader continuity was also a positive aspect of supervision that allowed the World Bank to build good rapport and communication with government agencies. Stakeholders’ general view was of good coordination between the World Bank and other development partners, mainly with USAID regarding the family medicine component and with WHO regarding support for the development of NHA. The World Bank, and the HPIU, systematically reported on output and intermediate outcome indicators. While only an element of the EMP, neither the World Bank nor the HPIU reported on the way project hospitals disposed of hospital waste. 2.73 Overall Bank performance is rated satisfactory. Borrower Performance Government Performance 2.74 Government performance was satisfactory throughout the project cycle. The government was highly committed to health sector reform, gave high priority to the project, and as confirmed by interviews, had a productive and balanced dialogue with the World Bank. A series of government policies punctuated project implementation, supporting and reinforcing various aspects of project design. Among them, a decree establishing the free-choice enrollment of patients with family doctors, a PHC strategy with family medicine at its center, and decrees for the consolidation of health services into hospital networks in Yerevan and in the marzes. The latter was approved in 2006 and prompted the anticipated move to APL II. Government budget allocations to the health sector increased despite the onset of the global economic crisis, while counterpart funds were provided in a timely manner. Implementing Agency Performance 2.75 The performance of the Ministry of Health, as the implementing agency, was satisfactory. The Ministry of Health was highly committed to the attainment of project objectives and was perceived by stakeholders as experienced and able to conduct high- quality technical work. Leadership also remained constant, changing only once during 22 implementation. Similarly, HPIU’ s staff was perceived by stakeholders as experienced and qualified to properly conduct financial and managerial work, even above standards for the region. The HPIU monitored project performance regularly, although more effort could have been placed on monitoring waste disposal issues in project hospitals. While state inspection bodies were responsible for monitoring compliance of hospitals with the national standards, the HPIU could have flagged the lack of information in this area. 2.76 Overall borrower performance is rated satisfactory. Monitoring and Evaluation Design 2.77 The APL results framework was adequate to monitor progress and demonstrate achievement of the objectives. The first phase originally included 10 performance indicators and 14 intermediate outcome indicators, most with complete baseline and target values. Indicators reflected health system performance dimensions related to coverage of PHC, hospital efficiency and productivity, and structural quality. Measures of the impact of institutional strengthening interventions were output-level indicators. Intermediate outcome indicators tracked project outputs well (figure 2.1). APL I also included five triggers for moving to the second phase, which were well aligned with the results framework. Given concern about the gatekeeper role of family physicians in urban settings, the project could have monitored the proportion of retrained physicians that could effectively practice their new skills. Implementation 2.78 Results framework indicators were regularly collected and reported by the HPIU. Satisfaction surveys were also conducted at hospital networks in Yerevan and two participating marzes in 2008. About 1,420 health staff and 2,054 patients participated in the surveys in Yerevan and 300 staff and 227 patients participated in the two marzes. During implementation, some indicators were dropped as unrealistic due to the lack of funds to pursue the activity (among them, independent audits in hospitals), or the potential of the project interventions to have a direct effect (among them, decline in abortion rates). While the use of health services disaggregated by socioeconomic groups was claimed to have been monitored, its progress was not reported at the end of the project in part because of the low pace in improving use rates by the poor during implementation of APL I. Use 2.79 At several points during implementation the results framework was used as a management tool for decision-making during project implementation. First, preparations for the second phase of the APL were anticipated as a result of meeting the 23 trigger indicators, which were tracked regularly. Second, monitoring and evaluation (M&E) was useful in assessing the evolution of hospital efficiency and productivity gains and providing local authorities with timely data on the progress of mergers. Third, the rapid pace of retraining physicians and nurses in family medicine as well as the expansion in training capacity encouraged the government to request additional financing from the World Bank to further advance these activities and meet the remaining training needs. However, as discussed before, family physicians were less effective in managing a larger set of health conditions in urban areas due to the availability of specialists. 2.80 The quality of M&E is rated substantial. 3. Health System Modernization Project (APL II) Relevance of the Objectives and Design Objectives 3.1 The second phase of the APL was approved in 2007, overlapping with the first phase. As stated in the financing agreement of March 9, 2007, the objective of the project was “to strengthen the Ministry of Health’s capacity for more effective system governance, scaling up family medicine-based PHC and upgrading selected health care service delivery networks in the Selected Marzes to provide more accessible, quality and sustainable health care services to the population” (World Bank 2007b, 5). The additional financing of December 2010 did not change the core project objectives. 3.2 APL II focused on key performance dimensions of the health care system: access, quality, sustainability, and governance. Providing sustainable health care services to the population implies that the services are provided efficiently and that resources allocated to the health care sector are sufficient. Therefore, this PPAR will assess efficiency and public financing under the sustainability objective. Since the project development objectives were not explicit on which performance dimension was to be improved in primary or secondary health care, the efficacy section will discuss both PHC and hospital networks (see appendix B, table B.2). Like APL I, specific emphasis to improve use by vulnerable groups was diluted when comparing program and APL II objectives. Relevance of the Objectives 3.3 APL II objectives continued to be relevant to health sector needs and government priorities as in the first phase. At project entry, Armenia still faced low use of health services mainly due to high out-of-pocket payments, oversized and poorly maintained infrastructure in the regions, and excessive focus on curative rather than preventive care. 24 After three years of APL I implementation, use of PHC and hospital services was slowly improving, and hospital optimization was still under way. The ambitious reform agenda in the health sector needed continuity in strengthening PHC through the organizational model of family medicine, optimizing hospital networks outside Yerevan, and strengthening SHA capacity as a purchaser of health care services. 3.4 The objectives were also relevant to the country and World Bank strategies. Armenia’s Development Strategy 2025 featured a pillar focused on enhancing human capital through improved access to quality social services (including health), and a separate pillar focused on improving social protection by enhancing efficiency of existing systems. Objectives were aligned with the World Bank’s FY05–08 Country Assistance Strategy pillar for reducing nonincome poverty, which advocated for increasing social sector spending and implementing systemic social sector reforms. Likewise, the FY14–17 Country Partnership Strategy had a strategic engagement cluster on improving efficiency and targeting of social/health services. The World Bank portfolio aimed at focusing on improving access for the bottom 40 percent of the population. Yet, as in APL I, ensuring equitable access to health services was not included in the APL II development objectives. 3.5 The relevance of objectives is rated substantial. Design Components 3.6 As in the first phase, APL II had four components. 3.7 Component A: Family Medicine Development (Appraisal $4.7 million; Additional Financing $5.45 million [total estimate $10.15 million]; actual $9.87 million). This component was to support strengthening of institutional capacity to train well-qualified family physicians and nurses as first-line PHC providers and improve their working environment. It was to complete planned training and retraining of 1,650 physicians and an equal number of nurses to ensure 100 percent population coverage, based on a ratio of one team per 1,700–2,000 population. About 50 rural ambulatories were to be upgraded, and outreach activities conducted to promote community participation. 3.8 Component B: Hospital Network Optimization (Appraisal: $20.77 million; Additional Financing $17. 0 million [total estimate $37.77 million]; actual $43.23 million). This component aimed to support the implementation of optimization plans in eight marzes that had not been covered by APL I by upgrading selected hospitals and refurbishing them with modern medical, IT, and HWM equipment. This component was also to finance technical work for architectural design, and training in 25 hospital management, quality assurance, accountability and fiduciary management arrangements, and HWM. 3.9 Component C: Institutional Strengthening (Appraisal: $2.58 million; Additional Financing $2.24 million [total estimate $4.82 million]; actual $0.66 million). This component aimed to further strengthen Ministry of Health capacity for policy making, planning, regulation, human resources development, and M&E, for more effective system governance and control of noncommunicable diseases. It was also to support strengthening of the governance and management structures of health care facilities and the oversight function of marz administrative structures. Support was to be made available to strengthen SHA operations, and to improve costing of publicly financed services and reimbursement mechanisms. The State Medical University was to benefit from consultancy services to upgrade its medical curriculum, improve its teaching and training facilities, and introduce new technologies for continuous medical education. 3.10 Component D: Project Management (Appraisal: $1.57 million; Additional Financing $0.63 million [total estimate $2.2 million]; actual $1.79 million). This component provided institutional support to the Ministry of Health through a Health Project Implementation Unit (HPIU), which was to oversee implementing day-to-day project activities and M&E. The component was to finance annual financial audits as well as training and operating costs of the HPIU. The 2010 additional financing was to support the rehabilitation of merged hospitals and the construction of one new hospital. Relevance of Design 3.11 APL II core objectives and project components were kept from the previous phase, thus there was a logical and plausible link between planned activities and expected outcomes. The theory of change illustrated in figure 2.1 continues to be valid for this second phase. APL II continued expansion of the family medicine model through the retraining of physicians and physical improvements of PHC facilities. While APL I financial assistance supported implementation of the optimization plan in only two marzes, APL II focused on securing funding for all the remaining marzes. As in APL I, limited attention was given to the conducive environment for family physicians to practice their new skills because of the ready availability of specialists and cultural factors that affected health-seeking behaviors. 3.12 As in the previous phase, budget support operations complemented APL II and its additional financing. The 2005–07 Poverty Reduction Support Credit series funded health-related actions to improve health financing sustainability, the development of hospital governance plans, and the implementation of the program on the prevention and control of noncommunicable diseases. The second development policy operation in 26 2011 also sought to improve affordability of health services for the poor and vulnerable by reforming health financing, including the launch of performance-based contracting at PHC level, and the expansion of service delivery by strengthening noncommunicable disease interventions (see appendix C, table C.6). 3.13 The APL continued to be an adequate instrument for the second phase. The instrument allowed continuity and consistency with the reforms initiated during phase one. The phased nature of the APL enabled graduate implementation of the program. The second phase continued the training of physicians and nurses, upgraded infrastructure of PHC facilities, seeking countrywide access to primary care. Similarly, the second phase expanded the geographical scope of the hospital optimization efforts. On the institutional side, the second phase supported the implementation of policies developed during the first phase. 3.14 Momentum in hospital optimization reforms and the government’s willingness to scale up its efforts led to the approval of APL II one year ahead of schedule. In just two years, APL I showed a good implementation record, particularly in hospital modernization. The government had also shown a record in implementing hospital mergers and networks effectively, as well as commitment to optimize health facilities in the marzes outside Yerevan on a much larger scope. The World Bank decided to move forward to the second phase to accelerate implementation of reforms that otherwise would be delayed. The demonstration effect of results from the first phase proved decisive for scaling up and the sustainability of results. However, the fast transition from APL I to APL II may have limited opportunities for course corrections in the implementation of the family medicine model in urban areas. 3.15 Most of the trigger indicators were fully met, which was positive considering only two years had passed since project approval (see appendix C, table C.7). An additional financing to APL II was considered at this point, but it was discarded given the scope of reforms being pursued, the size of additional investment needed, and the positive political climate toward a new operation. The second phase, with the accompanying additional financing, was completed as planned. By project closing, the APL instrument had allowed the World Bank to support Armenia’s health reform efforts for over a decade. 3.16 The relevance of design is rated substantial. 27 Implementation Key Dates 3.17 The project was approved on March 8, 2007, became effective on June 6, 2007, and closed on February 29, 2016, three years and two months later than scheduled. The project was restructured three times and the closing date extended twice. On March 19, 2010, a level 2 restructuring added activities for rehabilitation of merged hospitals, construction of a new hospital, and provision of medical equipment. On December 20, 2010, a level 1 restructuring and additional financing added activities for new hospital investments, revised the results framework to include new outcome targets, and extended the project closing date from December 31, 2012, to December 31, 2014, to accommodate the new investment plan. On March 26, 2014, a level 2 restructuring extended the project closing date from December 31, 2014, to February 29, 2016, for completion of civil works in hospitals. Planned versus Actual Expenditure by Component 3.18 Total costs of APL II were estimated to be $29.62 million, with 66 percent of the funds allocated to component B (Hospital Network Optimization). The additional financing was distributed across all components of APL II raising total project costs to $54.94 million. Exchange rate fluctuations explained differences with actual project costs ($55.59 million). The costs for component A (Family Medicine Development) were 26 percent higher than originally estimated because the project renovated more health facilities than planned. In contrast, the costs of components C (Institutional Development) and D (Project Management) were 87 and 23 percent smaller than envisaged due to the government’s decision to finance NHA and HSPA reports out of public funds, and not to use project funds for technical assistance and consultancy services to concentrate resources in infrastructure. 3.19 The project was originally financed by a $22 million IDA credit, supplemented by additional financing of $19 million through a loan from the International Bank for Reconstruction and Development, approved in December 2010. At project closing, IDA and the International Bank for Reconstruction and Development financed 75 percent of project costs, as envisaged. The borrower and local communities initially committed $7.62 million: $7.17 from the government, $0.3 million from the State Medical University, and $0.15 million from local communities. An additional $6.32 million of counterpart funds were added at the additional financing stage, for a total planned commitment of $13.94 million. The actual total contribution was $13.89 million, with $13.51 million from the government and $0.38 million from local communities, while the planned contribution from the State Medical University was not made (see appendix A, table A.7). 28 Implementation Experience Safeguards Compliance 3.20 Project design continued to integrate safeguard measures correctly. As in the first phase of the program, APL II was rated B in the environmental category and triggered safeguard policy Environmental Assessment (OP/BP 4.01). The potential harmful environmental effects from the construction activities and future operations of health facilities were qualified as limited in scope and severity. The EMP prepared for APL I was reviewed, and it continued to be valid with the introduction of minor updates. Most of the mitigating measures related to the construction and operation of facilities supported by the project were implemented, and draft national guidelines for HWM were finally adopted by Decree No. 03-N in 2008, with continued World Bank support during APL II. The ICR states that “To date, compliance with the EMP has been satisfactory” (World Bank 2016, 9). While hospital capacity for HWM was improved, some concerns remain regarding adequate handling of waste particularly in health facilities located far away from the capital where waste treatment companies regularly operate. Financial Management and Procurement 3.21 Financial management and procurement continued to be robust during APL II. As in APL I, the HPIU satisfactorily performed all financial management activities. Moreover, it managed to execute a substantial amount of project resources as infrastructure increased after additional financing. The borrower’s cofinancing was timely, and audit reports were publicly disclosed. Procurement was also satisfactory. The procurement unit of the HPIU had specialized staff who managed acquisitions effectively (civil works, purchase of medical equipment and supplies, furniture, and consultancy services) through different procurement methods in accordance with World Bank guidelines and regulations (World Bank 2016). Achievement of the Objectives Objective 1: Provide More Accessible Health Care Services to the Population Outputs 3.22 APL II continued to support improvements in PHC access and structural quality through material and human resources. The project financed the ongoing implementation of a one-year specialized training and retraining program in family medicine. By the end of the project, 1,676 family physicians and 1,804 nurses were trained. This represents about 95 percent of PHC professionals. The number of physicians was slightly below the target because doctors near retirement age did not 29 receive training. APL II strengthened PHC infrastructure: 112 health facilities in eight regions were constructed or renovated and equipped, exceeding the revised target of 100, including PHC facilities in 50 rural communities. 3.23 Hospital optimization was carried out in one network per marz, upgrading physical infrastructure of medical centers. According to interviews, 13 hospitals were renovated, two medical centers were newly constructed, representing about a third of hospitals and serving half the population outside of Yerevan. The IEG mission corroborated the substantial improvements of the medical center in Ararat marz. The Ararat Medical Center comprises the hospital, which was fully renovated with project funds, and the polyclinic for ambulatory care from Soviet times, which was not supported by the project. Figure C.7 in appendix C shows the big gap in physical capacity among these two facilities and illustrates the importance of the project investments. Including management professionals in hospitals, 4,118 total health personnel received training, exceeding the target of 3,700. Outcomes 3.24 The project helped the government improve access to and use of PHC services, although concerns remain about the gatekeeper role of family physicians. By 2016 and as of today, almost all of Armenia’s population is enrolled in and has access to a PHC facility. The strengthened PHC network benefited about 627,000 people in 2016. Per capita PHC visits increased from 2.4 in 2005 to 4.1 in 2017 (see appendix C, table C.9). Regional data show that all marzes supported by APL II registered a substantial increase in the number of per capita ambulatory visits (see appendix C, figure C.3). Yet the challenges of practicing family medicine persisted due to the proximity of specialists as well as cultural factors. In PHC facilities patients should see the family doctor first, being referred to a specialist only if necessary. Still, in 2016, 36 percent directly visited a specialist. The coexistence of family doctors and specialists in the same urban health facility limits the ability of the former to provide specialized care to the population on their own, and the full application of the new knowledge and skills obtained during the family medicine trainings. The habit of seeing a specialist directly is especially evident in public health centers, and polyclinics where 73 percent and 77 percent of the patients bypass the general physician, respectively. By comparison, 86 percent and 83 percent see the family doctor first in health posts and rural ambulatories, respectively. The main reasons for this care-seeking behavior are that: (i) the patient thought there was a need to see the specialist (25.8 percent); (ii) the patient did not trust the general physician (24.4 percent); (iii) others advised the patient to see the specialist (19.4 percent); and (iv) a district therapist or family doctor was absent (15.1 percent; Armenia, Ministry of Health 2016). 30 3.25 The project helped the government improve access and use of hospital services. The number of people receiving services through renovated hospitals exceeded the target of 2 million. The increase in hospital use rates was countrywide but it was more pronounced in district-level hospitals in marzes than in Yerevan. The marzes of Lori, Shirak, and Syunik were those experiencing larger increases during the project period (see appendix C, figure C.4). This may suggest a higher patients’ perceived quality at local level. Figure C.5 in appendix C shows that the number of overall marzes hospital admissions significantly increased between 2006 and 2015. 3.26 Achievement of objective 1 is rated substantial. Objective 2: Provide More Quality Health Care Services to the Population Outputs 3.27 The project contributed to improved preservice education for physicians, beyond other structural quality improvements in infrastructure and in-service training mentioned above. The State Medical University updated its curriculum, pedagogical methods, and student test system according to the European Union standards. 3.28 Project efforts to introduce modern hospital management tools for increasing governance and transparency were partially implemented. The main reasons were financial constraints, legislation, and political economy factors in the hospital sector that reduced the leverage of the Ministry of Health and the project to introduce supervisory committees, independent audits, and Public Performance Reports as planned. As discussed before, even though supervisory committees were established in each hospital, hospital budgets were insufficient to cover remunerations for committee members, thus deteriorating their functioning and sustainability. The implementation of independent audits was limited due to legislation establishing them as mandatory only when the annual revenues or balanced value of assets for a company exceed $2 million (dram 1 billion). Despite this, 7 of 13 hospitals prepared independent audits: 2 of them did so in compliance with mandatory legislation (Hrazdan Hospital in Kotayq marz and Gyumri Hospital in Shirak marz); while the other five (Ararat, Aparan, Gavar, Ijevan, and Alaverdi) underwent independent audits, voluntarily using local resources and technical support from the project HPIU. Regarding political economy factors, even though hospitals account for a large share of the health budget, their accountability to the Ministry of Health and the public is not strong because they are State Closed Joint Stock companies governed by strong stakeholders. So, like the other management tools, Public Performance Reports were not implemented due to their costs and that regulations did not enforce them. Advances in licensing and accreditation did not materialize. 31 Outcomes 3.29 Key health sector quality indicators improved. Process indicators suggest improvements in the quality of care. The increased burden of noncommunicable diseases and the achievements in structural quality encouraged the government to continue strengthening noncommunicable disease prevention and control by expanding the screening measures for hypertension, diabetes, and cervical cancer, and introducing incentive payments at the PHC level. Project data show the share of patients with hypertension who had at least one electrocardiogram increased from 42 percent in 2010 to 55.2 percent in 2014 and those with ischemic heart disease who had at least one total cholesterol test annually also increased from 33 percent to 54 percent in the same period. Table C.10 in appendix C also shows improvements in detection rates for common childhood conditions during preventive care examinations. 3.30 Improvements in health facility infrastructure and enhanced physician skills were expected to positively affect the perceived quality of care among the Armenian population. Based on HSPA surveys, positive perceptions of care quality rose both at PHC (from 64 percent in 2007 to 73 in rural areas and 64 percent in 2012 in urban areas) and hospital levels (from 64 percent in 2007 to 67 in rural areas and 74 percent in 2012 in urban areas). Satisfaction surveys conducted by the project in eight hospitals also show that for a sample of 50 patients, 92 percent were satisfied with the physicians, 94 percent rated the quality of health care services provided as good or excellent, and 98 percent rated the facility conditions as good or excellent. Most of surveyed health personnel were also satisfied with the equipment and facilities (78 percent) and pointed out the high qualifications of their colleagues (80 percent). 3.31 More recent HSPA data on the responsiveness of the health system also offer positive results on the extent to which the system promptly met people’s expectations. As expected, there was a significant increase in positive perceptions of patients about the state of hospitals’ basic amenities in Yerevan (from 76 to 92 percent) and even more pronounced in marz cities (from 67 to 90 percent) between 2012 and 2015. Other domains related to attitudes toward the patient, such as dignity, communication, autonomy, and confidentiality, have remained relatively constant, ranging from 75 to 90 percent (see appendix C, figures C.9 and C.10). A possible explanation is the waiting list to receive state-funded health services, which tends to postpone care for a few months. According to key informants, waiting lists are a major quality concern that derived from the global budget allocated to each hospital per state-funded program, which creates an artificial monthly quota of the health services that can be provided and reimbursed. This is not the case for emergencies or other services not covered by the BBP for which patients pay out of pocket. 3.32 Achievement of objective 2 is rated substantial. 32 Objective 3: Provide More Sustainable Health Care Services to the Population Outputs 3.33 In the postsocialist context, sustainability of health services called for an increase in the capacity and use of more cost-effective primary and preventive care, while reducing inefficiencies in the provision of secondary care that creates health budget savings to improve the quality of health care. In this sense, project efforts to implement the family medicine model and the rationalization of hospital networks, to increase the share of public expenditures in PHC, and to strengthen preventive services related to noncommunicable diseases were going in this direction. 3.34 The rationalization of hospital networks addressed overcapacity in the system. The project helped in implementing the government’s hospital master plan, which sought the consolidation of 24 existing hospitals into 10 hospital networks. The number of beds in 14 hospitals supported by project interventions decreased from 1,640 to 1,035 (see appendix C, table C.14); while the square meters of capacity for all regions decreased by 85 percent as planned. Public hospitals completed the transformation of their governance structure; however, routine use of supervisory committees and Independent Auditing practices for improved management, transparency, performance, and efficiency fell short of what was planned. 3.35 Sustainability of service delivery also required sufficient public financing. Public health spending as a percentage of GDP remained constant during the APL II at 1.6 percent (see appendix C, table C.2) despite the 2009 financial crisis and its aftermath. The proportion of the health budget allocated to PHC increased from 36.4 percent in 2006 to 37.2 percent in 2016, and the project supported improvements in preventive health services aimed at reducing the burden of relatively expensive care for late diagnosis with chronic disease. The recently approved medium-term expenditure framework for 2019–21 projected increases in health sector financing over the next three years. The budget allocated to health is expected to increase from dram 82.3 billion ($171.5 million) in 2018 to dram 106.6 billion ($222.1 million) in 2020, and dram 117.2 billion ($244.2 million) in 2021. However, based on GDP projections, the public health budget will continue to be about 1.5 percent of GDP by 2021 (World Bank 2018b). Outcomes 3.36 The use of preventive and cost-effective health services to contain the growing burden of noncommunicable diseases increased. As mentioned before, the increased attention to prevention and control of noncommunicable diseases increased the use of screening tests for such conditions. 33 3.37 As in APL I, but at the regional level, hospital optimization led to efficiency and productivity gains in marz hospitals. Marz hospitals supported by the project experienced a larger reduction in ALOS from 7.7 days in 2006 to 5.8 days in 2016 (25 percent) compared with the country as a whole (19 percent). Average occupation per hospital bed continued to progress from 201 to 240 days per year during 2007–16, reflecting a more efficient use of hospitals capacity after mergers (see appendix C, table C.10). Figure C.6 in appendix C shows the negative trend of bed capacity along with the positive trend in number of discharges. While there is no evidence on the extent to which net savings were achieved by mergers, due to a lack of a detailed analysis at hospital level, maintenance costs resulting from new and modernized infrastructure were estimated to represent less than 2 percent of the public health budget (World Bank 2018b). 3.38 Achievement of objective 3 is rated substantial. Objective 4: Strengthen Ministry of Health Capacity for More Effective System Governance Outputs 3.39 APL II provided technical assistance to strengthen administrative, analytical, policy making, planning, regulatory, and M&E capacity. The project continued to support the HSPA and NHA analytical tools to provide information for evidence-based policy decision. Reports were produced annually, and data were instrumental in policy decisions. The SHA received support to strengthen its purchasing capabilities and conducting costs analysis in the health sector. Outcomes 3.40 A culture of evidence-based impact assessment was established through the institutionalization of documents that monitored health policy—such as the HSPA and NHA reports. HSPA served to highlight the growing burden of noncommunicable diseases leading the government to strengthen screening preventive services for hypertension, diabetes, and cervical cancer at PHC level. The experience of producing the HSPA and NHA reports, as well as their findings, contributed to evidence-based policy decisions and to more effective system governance. Since 2016 a new methodology of health accounts was introduced, which resulted in the adjustment of 2014–2015 NHA statistics and subsequent estimates of health expenditures. Out-of- pocket spending under the new methodology rose to unrealistically high shares (about 80 percent), which casts doubt on the robustness and consistency of estimates according to some informants. 34 3.41 The project contributed to the introduction of necessary adjustments to health financing mechanisms. Examples of these are the introduction of global budgets in hospitals as a policy measure of cost containment, the adjustment of case reimbursement costs to regional hospitals, and the introduction of performance-based payments to incentivize screenings for early detection and prevention of noncommunicable diseases. 3.42 However, the institutional development component lost traction. Activities for institutional strengthening spent only 14 percent of the original allocated plus additional financing funds due to the government’s decision in 2011 to give priority to infrastructure investments, instead of using project funds for technical assistance and consultancy services. Therefore, some activities were discontinued or canceled, including providers’ payments based on disease-related groups, and management of equipment and maintenance. NHA and HSPA reports were institutionalized and financed out of public funds. 3.43 Achievement of objective 4 is rated modest. Efficiency 3.44 No quantitative evidence exists on the allocative efficiency of project investments. Unlike APL I, economic analysis at appraisal and completion focused on health system efficiency as well as a fiscal impact analysis, without a traditional cost- benefit analysis or financial analysis from the perspective of hospital balance sheets. 3.45 However, qualitative efficiency analysis offers rather positive results. APL II supported improvements in the use of preventive health care services, intended to reduce the burden of relatively expensive care for late diagnosis of chronic diseases. Also, in the postsocialist context, rebalancing primary and hospital care to reduce dependence on relatively expensive inpatient care is an efficient strategy. The project addressed the strengthening of primary care through conversion of existing specialists and training of new medical graduates, as well as the upgrading of rural primary care facilities in areas that had been underserved. Implementation efficiency was strong as the HPIU performed well in relation to fiduciary and procurement activities, while managing to execute a substantial amount of project resources as infrastructure investments increased after additional financing. 3.46 The limited practice of the family medicine model in urban areas that continued during APL II is a shortcoming in the efficient use of project resources. However, a crude analysis suggests that these inefficiencies accounted for only 6.5 percent of actual project costs, since the family medicine component used 18 percent of project resources, from which about 37 percent were allocated to urban areas according to population estimates. 35 3.47 The overall efficiency rating is substantial. Ratings Outcome 3.48 The overall outcome rating is satisfactory. As for APL I, the relevance of objectives and design of APL II are rated substantial since development objectives were well aligned with country needs and priorities, and project activities were in line with the previous phase. Objectives related to improving access, quality, and sustainability of health care services were substantially achieved, but institutional strengthening goals were modestly achieved. Efficiency is rated substantial. Risk to the Development Outcome 3.49 Risk to development outcome is rated moderate. The fiscal impact analysis arrived at an optimistic conclusion about future public health expenditures based on assumptions of a precrisis scenario in which the economy was growing at double-digit rates, higher use rates of health services, expansion of the BBP, increases in health staff salaries, and reductions in out-of-pocket spending. The BBP cost for 2015 was expected to be 3 percent of GDP (representing 14 percent of total government expenditures), while total public spending in health turned out to be 1.6 percent of GDP (and about 6 percent of total public expenditures). Ex post replication of fiscal impact analysis used real salary increases for doctor and nurses, which were considerably lower than anticipated. The effects of the global crisis implied a significant contraction of the economy (-14 percent). While investments in infrastructure may require significant maintenance expenditures on hospital balance sheets, they would represent less than 2 percent of the public health budget. The medium-term expenditure framework for 2019–21 projects increases in health sector financing over the next three years, but this financing continues to be projected at about 1.5 percent of GDP by 2021 (World Bank 2018b). Additional financing is thus critical for Armenia to improve its service delivery and to ensure financial risk protection. The World Bank continues to support the health sector through a $35 million Disease Prevention and Control Project, which focuses on improving maternal and child health services, strengthening prevention and management of selected noncommunicable diseases, and enhancing the efficiency and quality of selected hospitals. The upcoming Country Partnership Strategy envisages further health support. 36 Bank Performance Quality at Entry 3.50 Bank performance on quality at entry was satisfactory. The second phase built on the successful experience of APL I, and had strong synergies with a concurrent development policy loan. The World Bank worked in close coordination with other agencies such as WHO and USAID, which also provided technical assistance for the development of NHA and HSPAs, and supported health financing efforts to strengthen purchasing capacities of the SHA, respectively. The APL financing instrument continued to be adequate for implementing the needed reforms, and most of the trigger indicators were fully met only two years after the beginning of APL I. The World Bank team took advantage of the instrument flexibility and accelerated the preparation of the next APL II to opportunistically support the government’s decision to quickly move ahead with the optimization program in the marzes to seize the positive political momentum. However, this anticipation may have limited the World Bank in integrating lessons from the first phase into APL II design such as improving the performance of the family medicine model in urban settings. Quality of Supervision 3.51 Bank performance regarding quality of supervision was satisfactory. The World Bank continued to adequately monitor project performance and maintained close and productive communication with government counterparts. Fluid communication during the implementation period allowed the World Bank to react to issues and changing circumstances. The project was restructured to commit additional funds for more rehabilitation work and include other hospitals in optimization plans, while the results framework was improved to add missing baselines and targets. Monitoring and reporting of safeguard implementation was adequate, except for reporting on the state of hospital waste disposal in relevant project sites, which was identified as an area of concern in the completion report of the first phase. Many interviewees highlighted the role and support of the World Bank’s health team when the government decided to reallocate project funds into infrastructure limiting technical assistance resources to support the HPIU. 3.52 Overall Bank performance is rated satisfactory. Borrower Performance Government Performance 3.53 Government performance was satisfactory throughout the project cycle. Government support to the health reforms and the project continued to be high during the second phase of the program and counterpart financing was provided in a timely 37 manner even during the global financial crisis. As with APL I, reforms in various project areas were implemented, including the approval of regional hospital optimization plans and the issuance of hospital waste management guidelines. However, while a decree regulating the functions of Hospital Supervisory Committees was issued in 2005, the full spectrum of functions of the committee remain undefined by the end of APL II, which affected project implementation in this area. Building on the work done during APL I regarding HSPA and NHA, the government institutionalized the production of these reports, which were central to policy formulation and regarded as an important project achievement by stakeholders. Implementing Agency Performance 3.54 Implementing agency performance is rated satisfactory. The Ministry of Health continued to perform satisfactorily, accompanied by the HPIU support to financial, procurement, and supervisory work, both highly regarded by stakeholders. The HPIU continued to adequately monitor progress on results and implementation of project components. However, lack of information regarding hospital waste disposal was not addressed or reported by the HPIU. 3.55 Overall borrower performance is rated satisfactory. Monitoring and Evaluation Design 3.56 The design of the results framework was robust to monitor progress and demonstrate achievement of the objectives. Given that APL II built on the previous phase and had similar objectives, the selection of indicators did not change substantially (see appendix C, table C.7). The project included numerous indicators (12 key performance indicators and 17 intermediate outcome indicators) with mostly complete baseline and target values. Yet HPIU monitored and systematically reported on them. No improvement was made in the indicators for institutional strengthening outcomes, although in this phase project commitments for technical assistance were reduced due to a government decision. Implementation 3.57 M&E indicators were regularly collected and were reported quarterly by the HPIU. As in the first phase, some indicators were dropped during the 2010 restructuring due to the lack of funds to pursue the activity (for example, implementing supervisory committees and issuing Public Performance Reports at the hospital), lack of a direct influence of the project (such as decline in abortion rates), or to difficulties in extrapolating a related budget line from the overall health sector budget (the budget allocation for noncommunicable diseases). Other targets were revised consistently with 38 the extended closing date. The HSPA and NHA reports were produced and published regularly. These consisted of analysis of health status and health service use based on routine administrative data, surveys such as the Integrated Living Conditions Survey, and customized modules on out-of-pocket informal payments, among others. Use 3.58 The project’s results framework served as a management tool for decision- making during project implementation. Regular and close monitoring of indicators progress allowed timely adjustments of the results framework as was evident in Aide Memoires and restructuring papers. Data collected on hospital efficiency and productivity gains were useful for the government to introduce changes in the optimization plans of the hospital networks in marzes. 3.59 The quality of M&E is rated substantial. 4. APL Program Achievements and Challenges 4.1 The APL program made important contributions to the health sector reform in Armenia during the period 2004–2016. 4.2 The World Bank has contributed to improving the quality of PHC services. Structural quality dimensions improved in terms of material and human resources. The expansion of the family medicine model for PHC enhanced the qualifications of physicians and nurses to treat health conditions that typically had been addressed by specialists in the past. By the end of the program, about 95 percent of PHC professionals were retrained in family medicine. Working conditions for physicians at PHC facilities also improved through the construction, rehabilitation, and provision of medical equipment that enabled them to apply their newly acquired skills. Physicians’ knowledge and infrastructure capacities plausibly resulted in better management of the primary care health conditions improving process quality. Detection rates for common childhood conditions during preventive care examinations as well as screening for early detection of chronic diseases improved, while evidence on perceived quality is mixed. 4.3 Better accessibility to PHC and upgraded capacities attracted more patients to primary care facilities for preventive services or when sick. Program investments in the PHC sector contributed to increase coverage of PHC services. Currently, the entire Armenian population is covered by retrained family physicians at the PHC level. Outpatient visits per person per year increased steadily, closing the gap with respect to other international benchmarks, but they still have not reached the level of Soviet times (see appendix C, figures C.1 and C.3). However, the gatekeeper role of family physicians is still challenged by the high supply of specialists in urban health facilities and cultural 39 factors associated with patient preferences. The merger of hospitals may have accentuated this issue. Beyond a few attempts to establish independent family medicine practices in the city, the APL program did not address this issue and continued to support the retraining of physicians in family medicine. 4.4 The APL program also contributed to improved quality, efficiency, and use of hospital services both in the capital Yerevan and in marzes. Hospital mergers upgraded physical infrastructure of medical centers and reduced the overcapacity of the hospital networks. Perceived quality of services was satisfactory to a large extent. Hospitals’ responsiveness to meet the expectations of patients, particularly regarding the state of basic amenities, increased in Yerevan and even more so in marz cities. Patient satisfaction regarding dignity, communication, autonomy, and confidentiality have remained relatively constant, possibly due to waiting lists to receive hospital services. Utilization of hospital services increased during the APL program as inpatient care discharges reverted their downward trend, closing the gap with respect to other international benchmarks and almost reaching the level of the Soviet era (see appendix C, figure C.2 and table C.11). The increase in hospital use rates was countrywide but more pronounced in district-level hospitals in marzes than in Yerevan after 2012 (see appendix C, figure C.4). Higher use combined with reductions of excess capacity resulted in higher efficiency and productivity of hospital care services as evident in increased bed occupancy rates and reduced ALOS. From the perspective of hospital balance sheets, however, it is not clear to what extent mergers resulted in net savings considering the expected higher maintenance costs of the upgraded infrastructure. 4.5 By the end of the program, equity in the use of health care had improved, but this is unlikely to be attributable to the program. APL program objectives emphasized the provision of health care services to the most vulnerable population. Although the objectives of neither APL I nor II kept this focus on the poor, by the end of program the World Bank reported that the use of health services by the poorest income quintile increased from 3.9 percent in 2010 to 5.3 percent in 2015 (outpatient) and 5.0 percent to 9.7 percent (inpatient). According to Demographic and Health Survey data, the gap in access to health care services between the rich and the poor improved between 2005 and 2016. For instance, the share of women delivering a child in public health facilities increased from 86 percent for the lowest wealth quintile in 2005 (99 percent for the highest) to 95 percent in 2016 (95 percent for the highest). About 12 and 13.4 percent of women in the poorest and richest quintiles, respectively, reported having to pay out of pocket for delivery in 2016. Access barriers to health care services lessened over time. The share of women reporting problems with access to health care services when they were sick declined from 89 percent in 2005 to 64 percent in 2016 (Armenia, NSS and MOH, and ORC Macro 2006; Armenia, NSS and MOH, and ICF International 2017). 40 Obtaining money for treatment remains the main problem. Since PHC services are free (except for drugs and some diagnosis), it is generally argued that high hospital prices limit financial access to those health services not covered by the BBP. In 2005, concerns about poor service quality were also important for not seeking care, but in the Armenia Demographic and Health Survey 2015-16, such a quality dimension is not reported and distance to the health facility is listed as the second main barrier to access. 4.6 The APL program contributed to strengthening health sector governance, monitoring, and decision-making to better manage public health threats. The Ministry of Health improved its capacity for the evaluation of health sector performance. The institutionalization of documents monitoring health policy—the HSPA and NHA reports—helped highlight the growing burden of noncommunicable diseases and prompted the government to strengthen screening preventive services for selected chronic diseases during the second phase of the program. Technical assistance efforts helped strengthen the management of public health expenditures and capacities of the SHA as a purchasing agency. In addition, the program support was key to developing new legislation on HWM for health facilities, which did not previously exist in Armenia, but enforcement could be difficult to achieve. Program efforts to introduce modern hospital management tools for increasing governance and transparency did not fully materialize because of financial constraints, legislation, and political economy factors in the hospital sector that reduced the leverage of the Ministry of Health and the project to introduce supervisory committees, independent audits, and Public Performance Reports as planned. The institutional development focus of the APL program lost traction over time, as evidenced by the decrease in the share of project funds allocated to this component due to the government decision to give priority to infrastructure investments. 5. Lessons 5.1 The APL program shows that an approach that exploits synergies and lessons from other World Bank engagements in the health sector is important for undertaking complex reforms and helping the government stay the course of reform. Complementarities across lending and knowledge instruments allowed the World Bank to engage in a range of health policy areas, including health financing, governance and stewardship of health authorities, and service delivery. 5.2 Macro and micro health policies need to be combined in a manner that the unintended consequences of policy changes are not overlooked. For example, recurrent adaptations of the Basic Benefit Package—changes in services covered, entitled population groups, and the pricing system supported by policy-based lending—created uncertainty for patients about the boundaries of the benefit package, increasing the risk 41 of informal payments and potentially undermining health care use. Similarly, the introduction of health financing policy changes, such as the global budgeting mechanism, while improving efficiency and cost containment from a macro perspective, may have had deleterious effects on some dimensions of quality (notably through the creation of waiting lists). 5.3 A shortened period between the approval dates of successive phases of an APL can limit the opportunity to incorporate lessons from previous phases into the design of new ones. The second phase of the operation was advanced to only two and a half years after the approval of the first phase. While this allowed the program team to seize the political momentum to implement hospital optimization plans in the marzes, it also limited the time to incorporate lessons from the first phase into the design of the second phase and introduce course corrections in the implementation of the family medicine model. By the end of APL, I, it was clear that such a model was less suitable in urban areas because of the availability of specialists within the same facility. Yet, because of the limited time, the second phase did not include design components supporting private PHC practices in cities. 5.4 In country contexts with strong social and cultural factors affecting uptake of health care services, supply-side and systemwide policy reforms need to be combined with demand-side interventions addressing the health-seeking behavior of patients. Increased use of PHC services, especially by the vulnerable, depends on the extent to which services are accessible and affordable, have a minimum level of perceived quality, and on cultural factors that affect health-seeking behaviors. While the APL program addressed the accessibility and quality of health care services, attention to patient perceptions and preferences was not explicit, and a considerable share of patients continues to self-refer to specialists due to the preconception that those specialists are more capable of treating certain conditions than generalists are, illustrating that social and cultural preferences take longer to change. 5.5 While investments in infrastructure are not enough for health system modernization, they can help ensure acceptance of the proposed organizational changes involving strong stakeholders in the hospital sector. The enthusiasm of regional health authorities to pursue hospital mergers was strongly associated with the promise of major investments in the marz hospital networks. In fact, the implementation of the program was particularly successful in those regions where it was followed up with major infrastructure investments in those networks. 42 1The cited documents are the project appraisal document of APL I and the Financing Agreement of APL II. Overarching program objectives were not explicitly stated in the Development Credit Agreement of APL I (World Bank 2004b). 2The first trigger sets a target for increasing government budget allocated to the health sector. This is an input indicator that reflects the availability of counterpart funds as well as the continuation of government commitment. 3Merger of Mkhitar Heraci Yerevan State Medical University Hospital; merger of Surb Grigor Lusavorich Medical Center; and merger of St. Marie Medical Center. 4 Hrazden and Ijevan Medical Center. 5 This potential public budget mismatch adds to the existing gap between the case reimbursement paid by the government and the market price of services, for which health providers must cover either by cross-subsidizing with other charged services or informal payments. Hospitals ’ financing gaps may also generate incentives for fraud in service reimbursement claims, favoring the reporting of more expensive treatments. The government’s approval of a copayment mechanism for certain services aimed at partly reducing this financing gap, while reducing informal payments and strengthening gatekeeping at PHC through penalizing self-referral. Akkazieva and Jowett (2013), however, find that the introduction of copayments was insufficient to reduce informal payments, and that a more complex package of measures, including meaningful remuneration for medical staff is necessary. 43 Bibliography Akkazieva, B., and M. Jowett. 2013. “An Evaluation of the Impact of Introducing Patient Co- Payments on Access to Health Services, Financial Protection, and Levels of Official Payments in Armenia.” Health Financing Policy Paper Series, World Health Organization Regional Office for Europe, Copenhagen. Armenia, Ministry of Health. 2015. Armenia: Health System Performance Assessment. Yerevan, Armenia: Ministry of Health. ———. 2016. Armenia: Health System Performance Assessment. Yerevan, Armenia: Ministry of Health. Armenia, NSS (National Statistical Service) and MOH (Ministry of Health), and ORC Macro. 2006. Armenia Demographic and Health Survey 2005. Calverton, Maryland: National Statistical Service, MOH, and ORC Macro. Armenia, NSS (National Statistical Service) and MOH (Ministry of Health), and ICF International. 2012. Armenia Demographic and Health Survey 2010. Calverton, Maryland: National Statistical Service, MOH, and ICF International. ———. 2017. Armenia Demographic and Health Survey 2015-16. Rockville, Maryland, USA: National Statistical Service, MOH, and ICF. Hakobyan, T., M. Nazaretyan, T. Makarova, M, Aristakesyan, H. Margaryants, and E. Nolte. 2006. “Armenia: Health System Review.” Health Systems in Transition 8 (6): 1–180. Rechel, B., E. Richardson, and M. McKee, eds. 2014. Trends in Health Systems in the Former Soviet Countries. Observatory Studies Series 35. London: World Health Organization. Richardson, E. 2013. “Armenia: Health System Review.” Health Systems in Transition 15 (4): 1– 99. Švab, Igor. 2006. Final Report Consultancy visit (Evaluation Retraining in Family Medicine in Armenia), Ljubljana, Slovenia, November 17, 2006 World Bank. 1997. Staff Appraisal Report Republic of Armenia Health Financing and Primary Health Care Development Project, Municipal and Social Services, Country Department IV, Europe and Central Asia Region, June 30. Washington, DC: World Bank. ———. 2004a. “Armenia—Health System Modernization Project.” Project Appraisal Document 27708, World Bank, Washington, DC. ——— 2004b. Armenia—Health System Modernization Project Development Credit Agreement, Conformed. Washington, DC: World Bank. ——— 2007a. “Armenia—Health System Modernization Project (APL II).” Project Appraisal Document 38149–AM, World Bank, Washington, DC. 44 ——— 2007b. Armenia—Health System Modernization Project (APL II) Financing Agreement, Conformed. Washington, DC: World Bank. ——— 2010. “Armenia—Health System Modernization Project.” Implementation Completion and Results Report ICR1650, World Bank, Washington, DC. ———. 2012. Mainstreaming Environmental Management in the Health Care Sector: Implementation Experience in India & A Tool-kit for Managers. 2 vols. Washington, DC: World Bank. ——— 2016. “Armenia—Health System Modernization Project (APL II).” Implementation Completion and Results Report ICR3703, World Bank, Washington, DC. ——— 2017. Future Armenia: Connect, Compete, Prosper. A Systematic Country Diagnostic . Washington, DC: World Bank. ——— 2018a. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evaluation Group. Washington, DC: World Bank. ——— 2018b. Armenian Health Financing System with Multiple Private Insurers. Washington, DC: World Bank. 45 Appendix A. Basic Data Sheet Health Systems Modernization Project (Credit 3920-AM; P073974) Table A.1. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percent of Financing ($, millions) ($, millions) Appraisal Estimate Total project costs 25.50a 29.36 115 Loan amount 19.00 20.52 108 Cofinancing 1.25 1.24 99 Source: Implementation Completion and Results Report. a. Increases in total project estimate and actual reflect the exchange rate fluctuation in the SDR to dollar exchange rate over the project life. Table A.2. Cumulative Estimated and Actual Disbursements Disbursements FY05 FY06 FY07 FY08 FY09 FY10 FY11 Appraisal estimate ($, millions) 4.00 13.00 17.00 19.00 19.00 19.00 19.00 Actual ($, millions) 1.03 3.85 12.03 15.21 16.95 19.61 20.52 Actual as percent of appraisal 26 30 71 80 89 103 108 Date of final disbursement: October 31, 2010 Source: Project portal Table A.3. Project Dates Event Original Actual Concept review 04/30/2003 04/30/2003 Negotiations 04/30/2004 04/30/2004 Board approval 06/10/2004 06/10/2004 Signing 07/30/2004 07/30/2004 Effectiveness 12/14/2004 12/14/2004 Closing date 06/30/2009 06/30/2010 46 Table A.4. Staff Time and Cost World Bank Budget Only Staff Time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY02 1 1.9 FY03 21 90.17 FY04 67 166.34 Total 89 258.41 Supervision or ICR FY05 45 82.62 FY06 41 89.61 FY07 41 78.1 FY08 23 36.67 FY09 12 18.21 FY10 11 25.76 FY11 5 30 Total 178 360.97 Note: ICR = Implementation Completion and Results Report. a. Including travel and consultant costs. Table A.5. Task Team Members Names Titlea Unit Lending Toomas Palu Senior Health Specialist ECSHD Monika Huppi Senior Human Development Economist ECSHD Silviu Radulescu Senior Health Specialist ECSHD Susanna Hayrapetyan Senior Health Specialist ECSHD Tamar Gotsadze Projects Officer ECSHD Edmundo Murrugarra Economist ECSHD Daniel Miller Health Specialist HDHNE April Harding Senior Health Economist HDHNE Supervision/ICR Johanne Angers Senior Operations Officer ECSH1 Alexander Astvatsatryan Procurement Officer ECSO2 Anne Anglio Senior Program Assistant ECSHD Enis Bariş Senior Public Health Specialist MNSHH Shiyan Chao Senior Economist (Health) ECSHD Olena Fadyeyeva Senior Operations Officer ECACA Tamar Gotsadze Health Specialist ECSHD Susanna Hayrapetyan Senior Health Specialist ECSH1 Nicole L. La Borde Office Manager MNSHD 47 Carmen F. Laurente Senior Program Assistant ECSHD Rohit R. Mehta Senior Finance Officer CTRFC Satik S. Nairian Program Assistant ECCAR Panagiota Panopoulou Economist (Health) ECSHD Owen K. Smith Economist ECSH1 Arman Vatyan Senior Financial Management Specialist ECSO3 Betty Hanan Implementation Specialist (Consultant) ECSHD Note: ICR = Implementation Completion and Results Report. a. At time of appraisal and closure, respectively. Table A.6. Other Project Data Borrower or Executing Agency Follow-on Operations Amount Operation Credit no. ($, millions) Board Date Health System Modernization Project (APL II) in Support of the 2nd Phase of the Health Sector CR 4267-AM 22.00 03/08/2007 Reform Program 48 Health System Modernization Project (APL II) in Support of the 2nd Phase of the Health Sector Reform Program (Credit 4267-AM and Loan 7987-AM; P104467) Table A.7. Key Project Data Actual or Current Appraisal Estimate Estimate Actual as Percent of Financing ($, millions) ($, millions) Appraisal Estimate Total project costs 41.00 41.86 102 Loan and credit amount 41.00 41.86 102 Source: Implementation Completion and Results Report Table A.8. Cumulative Estimated and Actual Disbursements Disbursements FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 Appraisal estimate ($, millions) 1.76 5.18 13.50 20.71 29.33 33.84 37.38 40.11 41.00 Actual ($, millions) 1.76 5.18 13.50 20.71 28.71 36.00 40.00 40.29 41.86 Actual as percent of appraisal 100 100 100 100 98 106 107 102 102 Date of final disbursement: June 30, 2016 Source: Implementation Completion and Results Report Table A.9. Project Dates Event Original Actual Concept review 01/08/2007 01/08/2007 Negotiations 02/05/2007 02/05/2007 Board approval 03/08/2007 03/08/2007 Signing 03/09/2007 03/09/2007 Effectiveness 06/06/2007 06/06/2007 Closing date 12/31/2012 02/29/2016 49 Table A.10. Staff Time and Cost World Bank Budget Only Staff time Costa Stage of Project Cycle (no. weeks) ($, thousands) Lending FY07 15.07 47,670 Total 15.07 47,670 Supervision or ICR FY08 16.41 22,906 FY09 16.35 28,514 FY10 22.83 52,693 FY11 27.25 34,710 FY12 20.26 51,298 FY13 11.14 46,726 FY14 11.3 28,415 FY15 10.68 33,292 FY16 12.41 25,274 Total 148.63 323,828 Note: ICR = Implementation Completion and Results Report. a. Including travel and consultant costs. 50 Table A.11. Task Team Members Name Titlea Unit Lending Enis Barış Practice Manager GHNDR Olena Fadyeyeva Senior Operations Officer LLIOP Tamar Gotsadze Consultant GHNDR Susanna Hayrapetyan Lead Health Specialist GHNDR Satik S. Nairian Program Assistant ECCAR Svetlana Georgieva Raykova Associate Operations Officer CASPM Supervision or ICR Susanna Hayrapetyan Lead Health Specialist GHNDR Wezi Msisha Sr. Operations Officer SACKB Johanne Angers Senior Operations Officer GHNDR Alexander Astvatsatryan Consultant–Procurement Specialist GGO03 Enis Barış Practice Manager GHNDR Garik Sergeyan Sr. Financial Management Specialist GG021 Arman Vatyan Sr. Financial Management Specialist GGODR Carmen F. Laurente Senior Program Assistant GEDDR Patricio V. Marquez Lead Health Specialist GHNDR Satik S. Nairian Program Assistant ECCAR Owen K. Smith Senior Economist GHNDR Armine Aydinyan Procurement Specialist GG003 Darejan Kapanadze Safeguard Specialist GEN03 John Malmborg Consultant GHN03 Tamar Gotsadze Consultant GHNDR Gabriel Francis Program Assistant GHN03 Note: ICR = Implementation Completion and Results Report. a. At time of appraisal and closure, respectively. 51 Appendix B. Methodology Evaluation Questions, Data, and Collection Methods This multiproject Project Performance Assessment Report (PPAR) seeks to shed light on the effectiveness, consolidation, and sustainability of the whole APL program investments for the improvement of health care services. While the field-based assessment sought to answer standard evaluation questions consistent with PPAR methodology guidelines (relevance, efficacy, efficiency, sustainability), it also tried to answer the following questions, which are important building blocks of the theory of change toward the achievement of project objectives: • The role of evidence-based and lessons learned in shaping the project design. • The coherence and synergies among various intervention modalities (reforms at PHC that affect hospital results) and World Bank instruments (other development policy lending with health components, ASA) that support a vision of health care reform. • The role of health financing and providers payment mechanisms reforms to enhance care management at primary health care and secondary level. • Measures undertaking to mitigate social risks arising from hospital optimization process. • Key features and configuration of the HPIU (composition, tasks, qualifications and technical knowledge, strengths, political support). • In light of the upcoming capacity strengthening evaluation, the PPAR sought evidence of the effectiveness of capacity building. The APL I introduced internationally known analytic tools, such as HSPA and NHA reports, with the aim of strengthening evidence-based policies and system governance. The first HSPA was produced in 2007 with an initial frequency of every two years, but then, given its importance, the government decided to fund it with its own resources every year. The ICR for APL II provides some examples of the impact on policy decisions of all the evidence gathered by the HSPA and NHA (such as introduction of performance-based financing; World Bank 2016, 11). This information was corroborated by interviews. The main data sources and collection methods used in this PPAR were the following: Review of external academic and policy literature (country PHC and Health Financing strategies; performance-based financing). 52 Portfolio Analysis in Health sector (project documents, World Bank Group strategies, other projects in health, Advisory Services and Analytics). Interviews with internal and external stakeholders (World Bank staff, government, donors, and beneficiaries, including health providers and health service users). Secondary data sources (WDI, HMIS, DHS) disaggregated by population subgroups and target municipalities where available. Site visits (1 or 2 secondary care hospitals in Yerevan). Table B.1. Evaluation Matrix for Project Development Outcome Data Collection Methods Review of Interviews and Portfolio academic and focus groups analysis in policy with Secondary data Indicator health sector literature stakeholders sources Site visits Relevance Relevance of YES YES YES objectives Relevance of YES (a) YES (a) YES (a) design Efficacy objectives Governance YES (a, c, e, f) YES (c) YES (c,f) YES (f) YES Access-use YES (e_) YES YES YES Quality YES (e_) YES (c) YES (c) YES YES Efficiency YES (e_) YES (d) YES YES Sustainability YES YES YES Efficiency Quantitative and YES YES YES qualitative efficiency 53 Table B.2. Identification of Objectives This Multiproject DCA/FA PAD ICRR PPAR APL I Program objectives: Program objectives: Program objectives: As in Implementation Completion and Results to improve the to improve the Report Review organization of the health organization of the care system to provide health care system to more accessible, quality provide more and sustainable health care accessible, quality and services to the population, sustainable health care in particular to the most services to the vulnerable groups; and population, in particular better manage public to the most vulnerable health threats. groups; and better manage public health threats. PDO: The objective of PDO: The objective of the PDO: ICRR used PAD Project is to support the the Project is to support objectives: the Borrower’s first implementation of the GOA 1. Expanding access to phase of the Program health reform program quality primary health through (i) expanding through (i) expanding care. access to quality primary access to quality health 2. Improving quality and health care; (ii) improving care, (ii) improving the efficiency of selected quality and efficiency of quality and effectiveness hospital networks. of selected hospital selected hospital networks; 3. Laying groundwork networks, and (iii) and, (iii) laying groundwork for effective health for effective health sector establishing capacity for sector policy making health policy policy making and and monitoring. making and monitoring. monitoring. APL II Program objectives: Program objectives: Program objectives: Program objectives: improving the improving the organization improving the improving the organization of the of the health care system organization of the organization of the health health care system to to provide more accessible, health care system to care system to provide provide more accessible, quality and sustainable provide more more accessible, quality quality and sustainable health care services to the accessible, quality and and sustainable health health care services to population, in particular to sustainable health care care services to the the population, in the most vulnerable services to the population, in particular particular to the most groups, and to better population, in particular to the most vulnerable vulnerable groups, and manage public health to the most vulnerable groups, and to better to better manage public threats. groups, and to better manage public health health threats. manage public health threats. threats. 54 This Multiproject DCA/FA PAD ICRR PPAR PDO: to strengthen the PDO: to strengthen the PDO: to strengthen the PDO: to strengthen the Ministry of Health’s Ministry of Health’s Ministry of Health’s Ministry of Health’s capacity for more capacity for more effective capacity for more capacity for more effective system system governance, scaling effective system effective system governance, scaling up up family medicine-based governance, scaling up governance, scaling up family medicine-based primary health care and family medicine-based family medicine-based primary health care and upgrading selected health primary health care and primary health care and upgrading selected care service delivery upgrading selected upgrading selected health care service networks in the Selected health care service health care service delivery networks in the Marzes to provide more delivery networks in the delivery networks in the Selected Marzes to accessible, quality and Selected Marzes to Selected Marzes to provide more accessible, sustainable health care provide more provide more accessible, quality and sustainable services to the population accessible, quality and quality and sustainable health care services to sustainable health care health care services to the the population services to the population population Note: APL = adaptable program loan; PDO = project development objective; PPAR = Project Performance Assessment Report. 55 Appendix C. Economic, Social, and Health Indicators Table C.1. Socioeconomic and Health Indicators for Armenia, 2000–17 Indicator 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Socioeconomic and Poverty GDP growth (annual %) 5.9 9.6 13.2 14.0 10.5 13.9 13.2 13.7 6.9 −14.1 2.2 4.7 7.2 3.3 3.6 3.2 0.2 7.5 Inflation, consumer prices (annual −0.8 3.1 1.1 4.7 7.0 0.6 2.9 4.4 8.9 3.4 8.2 7.7 2.6 5.8 3.0 3.7 −1.3 .. %) Population, total (in million) 3.1 3.1 3.0 3.0 3.0 3.0 3.0 2.9 2.9 2.9 2.9 2.9 2.9 2.9 2.9 2.9 2.9 2.9 Poverty headcount ratio at .. .. .. .. 53.5 40.1 30.2 26.4 27.6 34.1 35.8 35 32.4 32 30 29.8 29.4 .. national poverty lines (% of population) Health Status Life expectancy at birth, total 71 72 72 72 73 73 73 73 73 73 73 74 74 74 74 74 75 .. (years) Maternal mortality ratio (modeled 40 39 45 43 41 40 40 35 39 36 33 31 30 28 26 25 .. .. estimate, per 100,000 live births) Mortality rate, neonatal (per 1,000 15.7 14.9 14.2 13.5 12.9 12.2 11.6 11.1 10.5 10 9.5 9 8.6 8.1 7.7 7.4 7 6.7 live births) Mortality rate, infant (per 1,000 26.5 25.2 24 22.9 21.8 20.7 19.7 18.7 17.8 16.9 16.1 15.3 14.5 13.8 13.1 12.5 11.9 11.3 live births) Mortality rate, under-5 (per 1,000 30 28.5 27.1 25.8 24.5 23.3 22.1 21 20 19 18 17.1 16.3 15.5 14.7 14 13.3 12.6 live births) Service Coverage Immunization, DPT (% of children 93 94 94 94 91 90 87 88 89 93 94 95 95 95 93 94 94 94 ages 12–23 months) Immunization, HepB3 (% of one- 55 69 91 93 91 91 78 85 89 93 94 95 95 95 93 94 94 94 year-old children) Immunization, measles (% of 92 93 91 94 92 94 92 92 94 96 97 97 97 97 97 97 97 96 children ages 12–23 months) Births attended by skilled health 96.8 98.4 98.8 99.2 99.5 97.8 99.7 99.9 99.9 100 99.5 100 100 100 100 .. 99.8 .. staff (% of total) Note:.. = not available. 56 Table C.2. Health Financing Indicators for Armenia, 2000–15 Indicator 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Current health expenditure (% of GDP) 6.5 6.5 5.7 5.6 8.2 7.0 5.6 5.4 4.3 4.7 5.3 3.8 6.7 8.3 7.0 10.1 Current health expenditure per capita 40.5 45.4 44.6 52.2 98.1 115.6 120.6 170.9 174.3 140.4 169.4 133.2 247.3 317.9 281.3 366.0 (current $) Domestic general government health 15.8 22.2 22.2 22.8 16.9 20.9 28.4 29.7 34.4 38.6 32.1 44.1 23.0 17.9 21.4 15.9 expenditure (% of current health expenditure) Domestic general government health 1.0 1.4 1.3 1.3 1.4 1.5 1.6 1.6 1.5 1.8 1.7 1.7 1.5 1.5 1.5 1.6 expenditure (% of GDP) Domestic general government health 4.2 6.1 5.7 5.7 6.7 7.4 7.9 7.2 6.7 6.3 6.4 6.7 6.9 6.2 6.3 6.1 expenditure (% of general government expenditure) Domestic general government health 6.4 10.1 9.9 11.9 16.6 24.2 34.3 50.8 60.0 54.2 54.3 58.7 56.8 56.8 60.1 58.3 expenditure per capita (current $) Domestic general government health 23.9 37.9 38.2 45.1 56.0 70.2 89.1 107.0 108.6 114.5 110.9 116.9 117.9 118.1 126.4 140.6 expenditure per capita, PPP (current international $) Domestic private health expenditure 75.9 65.0 67.2 70.5 79.1 72.6 63.1 61.8 59.4 55.0 62.8 47.5 72.7 79.7 76.4 83.0 (% of current health expenditure) Domestic private health expenditure 30.8 29.5 29.9 36.8 77.6 83.9 76.1 105.7 103.5 77.2 106.4 63.3 179.8 253.4 214.9 303.8 per capita (current $) External health expenditure (% of 8.2 12.8 10.6 6.7 4.0 6.5 8.4 8.4 6.2 6.4 5.1 8.4 4.3 2.4 2.2 1.1 current health expenditure) External health expenditure per capita 3.3 5.8 4.7 3.5 3.9 7.5 10.2 14.4 10.8 9.0 8.7 11.2 10.7 7.7 6.2 4.0 (current $) Out-of-pocket expenditure (% of 74.4 63.4 65.2 68.6 77.9 72.4 63.0 61.7 59.2 54.7 62.1 46.3 71.3 78.2 74.7 81.6 current health expenditure) Out-of-pocket expenditure per capita 30.1 28.8 29.1 35.8 76.4 83.7 76.0 105.5 103.1 76.8 105.3 61.6 176.3 248.6 210.1 298.8 (current $) 57 Table C.3. Health Financing Indicators for Armenia and Comparators, 2000–15 Armenia MIC ECAa European Union OECD members Indicator 2000–07 2008–15 2000–07 2008–15 2000–07 2008–15 2000–07 2008–15 2000–07 2008–15 Current health expenditure (% of GDP) 6.3 6.3 5.0 5.1 5.5 5.5 8.5 9.6 10.2 11.7 Current health expenditure per capita (current $) 86.0 228.7 85.5 220.6 210.1 496.6 2,188.3 3,391.3 2,907.0 4,360.3 Domestic general government health expenditure (% 22.4 28.4 41.3 50.6 61.8 64.5 70.5 79.3 58.3 63.5 of current health expenditure) Domestic general government health expenditure (% 1.4 1.6 2.1 2.6 3.4 3.5 6.1 7.7 6.1 7.5 of GDP) Domestic general government health expenditure (% 6.4 6.5 — 9.1 — 10.0 13.1 15.7 15.1 17.2 of general government expenditure) Domestic general government health expenditure per 20.5 57.4 34.9 111.8 116.5 305.6 1,739.2 2,782.9 1,878.2 2,833.7 capita (current $) Domestic general government health expenditure per 58.4 119.2 95.1 216.9 282.5 589.7 1,700.2 2,693.5 1,842.2 2,745.8 capita, PPP (current international $) Domestic private health expenditure (% of current 69.4 67.1 57.8 48.5 38.0 35.2 22.2 20.7 39.2 36.5 health expenditure) Domestic private health expenditure per capita 58.8 162.8 46.7 106.3 69.0 166.2 613.6 726.5 1,313.6 1,629.9 (current $) External health expenditure (% of current health 8.2 4.5 1.0 1.0 0.4 0.3 — — — 0.0 expenditure) External health expenditure per capita (current $) 6.7 8.5 0.9 1.9 0.9 1.4 — — — 0.6 Out-of-pocket expenditure (% of current health 68.3 66.0 45.8 38.1 30.4 30.6 14.5 15.0 15.7 14.4 expenditure) Out-of-pocket expenditure per capita (current $) 58.2 160.1 38.7 83.8 63.6 152.3 320.2 508.9 458.2 627.8 Note: — = not available; ECA = Europe and Central Asia; MIC = middle-income country; OECD = Organisation for Economic Co-operation and Development. a. International Development Association and International Bank for Reconstruction and Development countries. 58 Figure C.1. Armenia and Comparators: Outpatient Contacts Per Person Per Year Source: European Health for All database (HFA-DB). Updated June 2018. Armenian data from National Health Information Analytic Center, Ministry of Health of Armenia. Figure C.2. Armenia and Comparators: Inpatient Care Discharges Per 100 Source: European Health for All database (HFA-DB). Updated June 2018. Armenian data from National Health Information Analytic Center, Ministry of Health of Armenia. 59 Table C.4. Armenia: Health Services Utilization by Income Quintile, 2004 Percent of all individuals Percent of the sick Uses preventive care Are sick Are treated Poorest quintile 2.0 18.7 45.5 Quintile 2 3.4 18.8 64.0 Quintile 3 4.1 18.1 71.0 Quintile 4 5.1 19.2 83.1 Richest quintile 6.5 20.8 94.2 Source: 2004 Integrated Living Conditions Survey (World Bank 2007b). 60 Table C.5. Official Development Assistance Commitments in Health, 2000–16 Commitment ($, millions) Donor 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2000–16 % Total All Donors, Total 7.9 7.3 2.2 6.0 23.1 11.6 16.4 31.4 7.2 9.9 6.3 11.3 12.0 34.8 3.6 12.5 0.4 204.0 100 DAC Countries, Total 7.6 4.1 2.1 5.9 5.1 11.4 9.7 6.3 7.0 1.9 5.5 3.7 0.8 0.6 2.5 2.5 0.1 77.0 38 Austria .. 0.0 0.0 .. 0.0 0.0 0.2 0.4 0.1 0.1 0.0 0.0 0.0 .. .. 0.0 0.0 0.9 0 Belgium .. .. .. .. .. .. .. .. .. .. 2.4 .. .. .. .. .. .. 2.4 1 Canada 0.2 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0.2 0 Czech Republic .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0.0 .. 0.0 0.0 0 Denmark 0.7 .. .. .. .. 1.7 .. .. 2.1 .. 1.2 .. .. .. .. .. .. 5.7 3 Finland .. .. .. .. .. .. 0.1 .. .. .. .. .. .. .. .. .. .. 0.1 0 France 0.0 .. .. 0.0 0.0 0.0 0.0 .. 0.6 0.3 0.0 0.3 0.0 0.1 0.2 0.0 0.0 1.7 1 Germany .. 0.0 0.1 5.4 0.2 .. .. 0.2 1.5 .. .. .. .. 0.0 0.1 0.0 0.0 7.7 4 Greece .. .. 0.0 0.1 0.2 0.1 0.1 0.1 0.1 0.5 .. 0.0 .. 0.0 .. .. .. 1.3 1 Ireland .. .. .. .. .. 0.2 0.2 .. .. .. .. .. .. .. .. .. .. 0.4 0 Italy .. .. .. .. 0.0 .. .. .. .. 0.0 0.0 .. .. .. 0.8 .. 0.0 0.9 0 Japan 4.8 4.1 .. 0.1 0.3 2.6 0.6 0.0 0.1 .. .. 0.0 0.0 0.1 0.0 0.0 0.0 12.6 6 Korea .. .. .. .. .. .. 0.0 .. .. .. .. .. .. .. .. .. .. 0.0 0 Norway .. 0.0 .. .. .. 0.4 0.4 .. .. .. .. .. .. .. .. .. .. 0.9 0 Spain .. .. .. .. .. .. .. .. .. .. .. .. .. 0.0 0.0 .. .. 0.1 0 Sweden .. .. .. 0.2 .. 0.0 .. .. .. 0.0 0.0 0.0 .. .. 0.0 .. .. 0.2 0 Switzerland .. .. .. .. 0.0 .. 0.1 .. .. .. .. .. .. .. .. .. .. 0.1 0 United Kingdom 0.0 .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. .. 0.0 0 United States 1.8 .. 1.9 .. 4.3 6.4 8.1 5.5 2.6 1.0 1.8 3.4 0.8 0.4 1.3 2.5 0.0 41.8 20 Multilaterals, Total 0.3 3.2 0.1 0.1 18.0 0.2 6.7 25.2 0.2 8.1 0.8 7.5 11.2 34.2 1.1 10.0 0.3 127.0 62 United Nations, Total 0.3 0.2 0.1 0.1 0.2 0.2 0.1 .. 0.1 0.1 0.0 0.6 0.0 0.1 0.4 0.3 0.3 3.2 2 UNICEF 0.3 0.2 0.1 0.1 0.2 0.2 0.1 .. 0.1 0.1 0.0 0.0 0.0 0.0 0.3 0.1 0.1 1.8 1 WHO .. .. .. .. .. .. .. .. .. .. .. 0.6 0.0 0.1 0.1 0.2 0.2 1.4 1 61 Commitment ($, millions) Donor 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2000–16 % Total World Bank Group, .. 3.0 .. .. 17.8 .. 2.9 25.1 .. .. .. 2.1 6.0 33.3 .. .. .. 90.1 44 IDA Gavi .. .. .. .. .. .. .. 0.1 0.1 0.4 0.8 0.2 0.7 0.8 0.7 0.4 .. 4.1 2 Global Fund .. .. .. .. .. .. 3.6 .. .. 7.6 .. 4.6 4.5 .. .. 9.2 .. 29.6 15 Source: Data extracted on Sep 2018 from OECD. Stat. Note: Data from Creditor Reporting System are collected on individual projects and programs. “Sector 120 I2. Health, total” was used to identify the commitments. A sector or main purpose category defines the main economic or social infrastructure categories which an individual activity is intended to foster. The sector classification also includes a number of categories which are not allocable by sector. These are general budget support; debt relief; humanitarian aid, emergency assistance; food aid; support to nongovernmental organizations and administrative costs... = not available; IDA = International Development Association; UNICEF = United Nations Children’s Fund; WHO = World Health Organization; Table C.6. Armenia: World Bank Health-Related Portfolio, 1996–2019 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Appro- Exit/Deli- Instru- Instr Prac- IBRD+IDA val FY very FY ment Proj ID Project Name Type/Prod tice Com. 1996 1997 Lending P042793 SAC DPL MTI 60.0 1998 1998 Lending P044796 SAC 2 DPL MTI 60.0 Health Financing and Primary Health IL HNP 10.0 1998 2004 Lending P050140 Care Development 1998 2003 Lending P051026 SATAC 2 IL MTI 5.0 1999 2001 Lending P051171 SAC 3 DPL GOV 65.0 ESW MTI - 2000 2000 ASA P061069 Institutional and Governance Review 2001 2003 Lending P065189 SAC 4 DPL MTI 50.0 2002 2002 ASA P070384 PER ESW GOV - 2002 2002 ASA P072372 POVERTY STUDY ESW POV - 2003 2003 ASA P074933 CPAR–AM ESW GOV - 2003 2004 Lending P075758 SAC 5 DPL GOV 40.0 2004 2004 ASA P071257 CFAA–AM ESW GOV - 2004 2010 Lending P073974 HEALTH SYS MOD (APL #1) IL HNP 19.0 2004 2004 ASA P078940 POV ASSMT ESW POV - 2004 2004 ASA P085622 POV ASSMT FLWP TA NLTA SPL - 2005 2005 Lending P078673 PRSC DPL MTI 20.0 2005 2005 ASA P090352 PROG PER TA NLTA GOV - 2005 2005 ASA P090784 PROG POV ASSMT ESW POV - 2006 2006 Lending P093459 PRSC 2 DPL MTI 20.0 2006 2006 ASA P096944 PROG POV ASSMT ESW POV - 2006 2011 Lending P099832 AVIAN FLU–AM IL SURR 6.3 2007 2007 Lending P093460 PRSC 3 DPL MTI 28.0 2007 2007 ASA P096939 PROG PER Series 2 & 3 ESW MTI - 2007 2007 ASA P101610 PROG POV ASSMT ESW SPL - 2007 2016 Lending P104467 HLTH SYS MOD (APL II) IL HNP 22.0 2008 2008 ASA P107267 PROG PER Series 4 ESW MTI - 2008 2008 ASA P107789 POVERTY ASSESSMENT NLTA POV - 2009 2009 ASA P112826 CPAR UPDATE–AM ESW GOV - 2009 2009 ASA P113610 Poverty Monitoring TA NLTA POV - 2010 2010 ASA P116771 Programmatic Poverty ESW SPL - 2010 2017 Lending P117384 PSMP II IL GOV 9.0 2011 2011 Lending P116451 Armenia DPO 2 DPL MTI 25.0 2011 2016 Lending P121728 HSMP2–AF IL HNP 19.0 2012 2012 Lending P122195 AM-DPO 3 DPL MTI 80.0 2013 # Lending P128442 DISEASE PREVENTION & CTRL IL HNP 35.0 2015 2015 Lending P143040 ARMENIA DPO2 DPL MTI 75.0 2016 2016 ASA P151948 Armenia Public Expenditure Review ESW MTI - 2017 2017 ASA P159613 Armenia Universal Health Coverage NLTA HNP - 63 Table C.7. Results Framework Indicators, APL I and II Operationn # Indicator Type 1 proportion of population covered by qualified family medicine practices KPI 2 ALOS in project hospitals KPI 3 hospital productivity improvement KPI 4 health system performance report published KPI 5 national health accounts developed KPI public hospitals that use updated financial management and accounting 6 KPI procedures 7 public hospitals are supervised by effective supervisory committees KPI 8 public hospitals have independent audits KPI 9 public hospitals issue Public Performance Reports KPI 10 M&E system for HIV/AIDS strategy in place KPI APL I 11 number of certified family doctors retrained IOI 12 number of certified family nurses retrained IOI 13 health facilities constructed, renovated, and equipped IOI 14 abortion rates decline IOI 15 proportion of health budget allocated to PHC IOI 16 reduction of square meters of hospital space IOI 17 proportion of physicians licensed according to new procedures IOI percentage of SHA contract with health services providers are concluded no 18 IOI later than February 1 each year 19 management of SHA contracts satisfactory or better to health care providers IOI 20 proportion public hospitals with trained key management staff IOI 21 reliable data on risk behavior and effectiveness of interventions IOI 1 Health System Performance Assessment report issues once every two years KPI 2 National Health Accounts Report published annually KPI Public Hospitals in project sites have published financial audit reports form 3 KPI APL II independent audits Proportion of Armenian population covered by qualified family medicine 4 KPI practices Increased use of (a) outpatient services and (b) inpatient services by the 5 KPI poorest income quintile 64 Operationn # Indicator Type 6 Perceived quality of care in marzes increased (PHC and hospital) KPI 7 ALOS in marz hospitals decreased close to OECD average KPI Proportion of patients with (a) hypertension who had at least one ECG 8 annually and (b) ischemic health disease who had at least one total KPI cholesterol test annually 9 project beneficiaries (PHC and hospital) KPI 10 public hospitals that are supervised by effective supervisory committees KPI 11 public hospitals that issued Public Performance Reports KPI 12 budget allocation of the prevention and control of NCFs increased KPI 13 proportion of health budget allocated to PHC IOI 14 abortion rates decline IOI 15 number of certified family doctors trained IOI 16 number of certified family nurses trained IOI 17 number of new guidelines developed and disseminated IOI 18 number of health facilities constructed, renovated, and equipped IOI 19 health personnel receiving training IOI 20 reduction of square meters of hospital space IOI 21 all project hospitals apply updated environmental management guidelines IOI 22 increased number of admissions in project hospitals IOI 23 percentage of SHA contract with health services providers are concluded no IOI later than 30 days after budget approval in parliament 24 management of SHA contracts satisfactory or better to health care providers IOI 25 proportion of physicians licensed according to new procedures IOI 26 proportion of key management staff in project hospitals trained on financial IOI management after the results of performed independent audits 27 State Medical University has revised curriculum in line with European Union IOI countries 28 M&E system for NCD is in place IOI 29 reduction in out-of-pocket payments for essential health services IOI Note: ALOS = average length of stay; APL = adaptable program loan; KPI = key performance indicator; NCD = noncommunicable disease; PHC = primary health care; SHA= State Health Agency. 65 Table C.8. Triggers for Adaptable Program Loan Program Triggers as Defined by PAD Achievement of Triggers for APL II Meeting the yearly targets for health Fully Because several years of double-digit GDP growth sector public expenditures as set out in met exceeded PRSP projections, health budget as a share the PRSP (these will be updated in of GDP fell short of yearly targets. However, PRSP accordance to changes of these absolute budget targets were met or exceeded: indicators in PRSP): 2005: 1.8 percent of GDP and 8.6 percent 24,691 million of Armenian Drams [PRSP target: of budget expenditures 24,900] 2006: 1.9 percent of GDP and 9.2 percent 1.5 percent of GDP of budget expenditures 31,079 million of Armenian Drams [PRSP target: 30,800] 2007: 2.0 percent of GDP and 9.6 percent 1.6 percent of GDP (expected) of budget expenditures 39,355 million of Armenian Drams [PRSP target: 35,500] Meeting at least 75 percent of the Partially targets of time-bound action plan to met improve financial management and accountability of public hospitals: ensure that all public hospitals use Fully All public hospitals use updated financial management updated financial management and met and accounting procedures mandated by law. accounting procedures all public hospitals are supervised by Partially A new government decree (N-1187-N) defines the effective Supervisory Committees met roles, responsibilities, and staffing of Hospital Supervisory Committees. Most marz hospitals have their supervisory committees established. Yerevan municipality passed a decree (July 26, 2006) regulating the establishment and functioning of the Supervisory Committees for the Joint Stock Companies, which all public hospitals in Armenia are. Expected to be fully met by the end of 2007. accounts of at least 80 percent of public Partially 11 hospitals in Yerevan already independently audited. hospitals are independently audited met In marzes, 22 hospitals had independent audits in 2005. In percentage terms, 58 percent of all hospitals have so far had their independent audits (85 percent in Yerevan and 50 percent in the marzes). Expected to be fully met by the end of 2007. 20 percent of hospitals issue annual Pending While no hospital has yet been able to issue an annual Public Performance Reports performance report, three merged networks in Yerevan prepared their three-year strategic business plans as the basis for future performance reports. High level managerial training is ongoing, and first reports are planned to be issues in 2008. Expected to be met by the end of 2008. Successful implementation of the Partially As a result of three mergers at the time of APL II selected hospital network optimization met approval, total number of hospital contracts decreased projects under the first phase and from 124 in 2004 to 115 in 2006. realizing the efficiency gains: 66 average stay in these hospitals is no Partially The average length of stay in the three mergers was longer than 10 days met 9.1, 12, and 12.3. productivity of inpatient services Fully Bed occupancy rate increased in all three mergers in improves by at least 20 percent met more than a 20 percent: from 37, 56 and 42 percent in compared with 2003 baseline 2003 to 50, 70, and 59 percent in 2005. At least 40 percent of populations is Partially covered by family medicine practices met that are staffed by trained family doctors and Fully As of 2006, 47 percent of the total population covered family medicine nurses met by family medicine practices. managerially autonomous from specialist Fully In 2005, 266 ambulatories had become managerially care met autonomous from polyclinics. have independent contracts with the Partially In April 2006, a decree was passed to establish SHA met independent practices for greater managerial autonomy and the right to have contractual agreements with the SHA as of 2008. The first HSPA and first set of NHA will Partially have been issued and an updated met national health strategy drafted with clear performance goals the first set of NHA will have been issued Fully NHA completed, a draft report submitted to IDA. met the first HSPA will have been issued Fully Scope and purpose of the HPSA prepared and content met agreed. The preparatory work, including the design of the questionnaires for the upcoming survey that will provide the primary data for assessment, has been completed. Survey ongoing and the report will be submitted to IDA by Summer 2007. an updated national health strategy Partially The Ministry of Health has recently called for the drafted with clear performance goals met establishment of a committee to begin the consultations for the drafting of the national health strategy. Source: Prepared by the Independent Evaluation Group based World Bank 2004 and 2007b. Progress in triggers as reported in World Bank 2007b. Note: APL = adaptable program loan; GDP = gross domestic product; HSPA = Health Sector Performance Assessment; IDA = International Development Association; NHA = National Health Account; PAD = project appraisal document; PRSP = Poverty Reduction Strategy Paper. 67 Box C.1. The Adaptable Program Loan Instrument The World Bank introduced the adaptable lending program instrument in September 1997. Adaptable program loans (APLs) provided phased support for long-term development programs that involved a series of loans. The instrument was discontinued in 2012, but similar lending approaches have been introduced since. Instrument Choice APLs required agreement on (i) the phased long-term development program supported by the loan, (ii) sector policies relevant to the phase being supported, and (iii) priorities for sector investments and recurrent expenditures. Progress in each phase of the program was reviewed and evaluated, and additional analysis undertaken as necessary, before the subsequent phase could be initiated. APLs would be used when sustained changes in institutions, organizations, or behavior were key to successfully implementing a program. They could be used to support a phased program of sector restructuring, or systemic reform in the power, water, health, education, and natural resource management sectors, where time is required to build consensus and convince diverse actors of the benefits of politically and economically difficult reforms. Triggers Triggers are well-defined milestones for moving from one APL phase to the next one, linking funds disbursements with program implementation and progress in achieving the development objectives. Good practice in defining APL triggers proposed a limited number of clearly specified triggers that measured various aspects of project implementation, continued government commitment, and progress toward objectives. Meeting triggers was a condition for management approval of subsequent phases of the APL, with an avenue for continuation when triggers were not met. The Board could approve subsequent phases subject to the World Bank’s justification to wave this requirement, informed by an assessment of implementation progress and the relevance and feasibility of original triggers. Advantages and Disadvantages APLs worked well for client engagement that took a long-term perspective on sectoral reform, usually supporting an acknowledged government program. APLs built strong partnerships with government agencies and main stakeholders, continuing support to capacity built in initial phases. APLs allowed for learning opportunities to be incorporated during implementation and in subsequent phases. Government ownership was key to project success in many cases. In contrast, some disadvantages regularly identified in APLs were the rigidity of triggers and their diminishing relevance over time, both of which limited the flexibility of the instrument. Other factors identified were program complexity, lack of borrower readiness for implementation, and changing priorities that come with new governments. Compared with stand-alone projects, APLs did not lead to significant reductions in processing time or preparation costs for subsequent phases. Other loans in the series had the same requirements as the first, except for the option of approvals by the Board on an absence of objection basis. Demand for APLs declined as approval on an absence of objection basis for stand-alone 68 projects and additional financing—with more streamlined processes—became more commonly used. APL and the Multiphase Programmatic Approach Multiphase Programmatic Approach (MPA) is an approach, as opposed to an instrument. In fact, an MPA-supported program could involve the approval of a series of phased projects that use a combination of instruments including investment project financing (IPF), IPF guarantees, and Program-for-Results financing. Development programmatic lending was excluded from this approach. The MPA preserves the adaptability of the APL instrument, but introduces greater emphasis on lessons learning, heavily investing in monitoring and in incorporating feedback. A major difference is that the MPA removed the use of triggers, leaving to management the decision to commit additional resources for other projects under the MPA umbrella, subject to satisfactory project performance. With the MPA approach, management would request Board approval of the financing envelope to support the development objectives and causal chain for the entire program. The first phase approval process would be similar to other instruments. For subsequent phases, management would be authorized to commit only if the new phase was consistent with the Program Financing approved by the Board. Such authorization, combined with streamlined processing steps and simplified documentation, are expected to decrease project processing time for subsequent phases. Source: World Bank 2001, 2002, and 2017; IEG interviews. Box C.2. Hospital Waste Management in the EMP The Environmental Management Plan (EMP) for Adaptable Program Loan (APL) I flagged the potential adverse environmental impacts—proliferation of diseases and groundwater contamination—that could result from inadequate handling of medical waste, hazardous wastewater, waste gases, and spillages of hazardous material during operation of project- supported hospitals. At that point, the government of Armenia did not have the legal and institutional framework needed to regulate and supervise hospitals’ handling of waste. The EMP identified this gap and incorporated measures to mitigate these risks into project design. * The measures addressed the need for setting standards, responsibility, and awareness at a broader level, while other measures sought to provide the means and define responsibilities for adequate hospital waste management at the facility level. APL II updated and adopted the original EMP, which included plans to build the capacity for hospital waste management in the facilities added to the optimization program and to support the approval of national guidelines. Most EMP activities were implemented. As shown in table BC.2.1, national guidelines were developed, staff were trained, supplies and equipment were provided, and waste segregation was being implemented in all project hospitals. 69 Figure BC.2.1. Development of National Guidelines EMP Measures EMP Activities Included in Project Design Status of Implementation Developing Component C Operational Guidelines and Operational Guidelines and capacity and Strengthening government capacity Government capacity operational Government Development of Waste Draft national guidelines on guidelines for Capacity Management Regulations and Health Care Waste Management medical waste Guidelines. were developed and adopted by management Environment capacity building and Decree Number 03-N in 2008. in health care training program. Regional health management facilities staff was trained on the content, Public awareness campaigns supported by importance and implications of directed to the population to the project the guidelines. A training of sensitize them on the harmful effects of medical waste and to trainers’ course was conducted report unacceptable practices. for specialists from the State Hygienic and Anti-Epidemic Inspectorate. No information available on the realization of public awareness campaigns. Component B Hospital Capacity Hospital Capacity Hospital Provision of waste management Supplies and equipment for Network supplies and equipment management of medical waste Optimization Training of staff on routine and provided in all project hospitals. emergency procedures Training on management of Assign clear responsibility for medical waste imparted in all medical waste management in the project hospitals. A training of new management structure of trainers’ course was conducted pilot hospitals. (APL II only: for epidemiologists responsible responsibility to be assigned to a for HWM in hospitals. member of the executive Responsibility for HWM in management team) hospitals was assigned. Segregation of waste introduced inInterviewees mentioned that solid waste segregation is done by all upgraded clinics. (APL II only: plan of segregation of waste and trained hospital staff and it is organizational policies of waste done satisfactorily. By law all management introduced in all hospitals must contract with a upgraded clinics) licensed company for waste Licensed companies should collect collection. Project documents health care waste generated in don’t report this as an issue. facilities as appropriate Note: a. Good practice suggests that when the borrower has inadequate legal or technical capacity to carry out key functions, such as environmental monitoring, inspections, or management of mitigatory measures, the project should include components to strengthen that capacity (World Bank 2012). Source: World Bank 2002, 2004, 2007a, 2007b, 2010, and 2016; and Independent Evaluation Group interviews. 70 Table C.9. Armenia: PHC Outpatients Visits, 1980–2017 (number) Indicator 1980 1990 1995 2000 2005 2010 2015 2016 2017 Visits to PHC facility 27,816.4 27,930.0 18,121.5 7,803.6 7,731.0 11,595.6 12,247.5 12,421.6 12,103.6 (thousand) Ambulator visits 24,758.7 24,618.2 16,355.3 6,725.4 6,773.0 10,401.7 11,127.8 11,267.8 11,054.7 Home visits 2,252.5 2,419.8 1,397.2 737.3 963.2 807.3 623.8 646.2 577.8 Average number of 9.0 7.8 4.8 2.1 2.4 3.6 4.1 4.2 4.1 visits per capita (including PHC visits, home visits, and emergency) Average number of 7.6 4.7 2.0 2.3 3.4 3.8 3.9 3.8 visits per capita (including PHC visits and home visits) Source: National Health Information Analytic Center, Health and Health Care Yearbook. Figure C.3. Annual Ambulatory Visits Per Capita by Marzes, 2006–15 Source: Armenia, Ministry of Health 2016. 71 Table C.10. Preventive Examinations of Children Ages 0–14 and Detected Conditions Detected With weakening With weakening With speech With disorders Number of hearing of eyesight defects With scoliosis of posture of % of % of % of % of % of examined exami- exami- exami- exami- exami- Year children Total ned Total ned Total ned Total ned Total ned 2004 616,865 1,030 0.17 8,029 1.3 2,994 0.49 279 0.05 4,211 0.68 2005 615,271 1,163 0.19 8,895 1.45 3,126 0.51 364 0.06 4,332 0.7 2006 584,583 1,035 0.18 9,908 1.69 3,539 0.61 375 0.06 4,109 0.7 2007 581,885 1,029 0.18 9,719 1.67 3,731 0.64 403 0.07 3,788 0.65 2008 577,542 1,042 0.18 10,638 1.84 3,575 0.62 510 0.09 4,024 0.7 2009 575,771 1,048 0.18 10,589 1.84 3,869 0.67 510 0.09 3,693 0.64 2010 574,060 973 0.17 10,308 1.8 3,520 0.61 656 0.11 3,136 0.55 2011 571,432 936 0.16 10,314 1.8 3,881 0.68 600 0.1 3,125 0.55 2012 574,943 905 0.16 10,980 1.91 3,843 0.67 613 0.11 3,312 0.58 2013 570,605 951 0.17 12,162 2.13 3,808 0.67 604 0.11 3,764 0.66 2014 568,171 952 0.17 12,372 2.18 3,562 0.63 605 0.11 4,046 0.71 2015 578,802 1,193 0.21 11,873 2.05 3,462 0.6 589 0.1 3,540 0.61 2016 584,708 1,217 0.21 12,503 2.14 3,457 0.59 568 0.1 3,649 0.62 Source: Armenia Health and Health Care Yearbook, 2017. 72 Table C.11. Armenia: Hospital Beds, Hospitalizations, and Bed Occupancy Hospital Beds Hospitalizations Per Days spent by the 10,000 Per 100 Bed occupancy patient on the bed Year No. Residents No. Residents (average no. days) (average no.) 2001 16,157 42.5 186,828 4.9 136 11.7 2002 13,968 43.5 197,365 6.1 153 10.9 2003 14,208 44.2 220,561 6.9 163 10.5 2004 14,259 44.3 235,008 7.3 169 10.3 2005 14,353 44.6 253,810 7.9 173 9.8 2006 14,276 44.3 269,546 8.4 178 9.4 2007 13,126 40.6 285,680 8.9 201 9.3 2008 12,358 38.2 306,635 9.5 223 9.0 2009 12,068 37.1 317,726 9.8 227 8.6 2010 12,160 37.3 323,962 9.9 223 8.3 2011 12,236 37.4 346,999 10.6 225 7.9 2012 12,241 40.4 375,316 12.4 236 7.7 2013 12,268 40.7 373,069 12.3 236 7.8 2014 12,514 41.6 406,552 13.5 246 7.5 2015 12,532 41.8 393,540 13.1 240 7.6 2016 12,493 41.8 399,734 13.4 240 7.6 Source: Armenia Health and Health Care Yearbook, 2017. Figure C.4. Hospitalization Rate Per Marzes Per 100 Population, 2006–15 Source: Armenia, Ministry of Health 2016. 73 Figure C.5. Hospital Admissions, Yerevan versus Marz Hospitals (absolute figures), 2006–15 (number) Source: Armenia, Ministry of Health 2016. Table C.12. Armenia: Service Utilization in Mergers Supported by the Project Full-time Equivalent per ALOS Bed Occupancy (%) 1,000 bed days Merger 2004 2007 2009 2004 2007 2009 2003 2008 2009 Merger B.1 9.6 8.6 7.9 41 46 69 10.8 9.7 8.8 Merger B.2 10.8 8.1 6.8 54 92 88 16.6 9.7 9.4 Merger B.3 10.6 7.9 7.7 53 60 71 14.7 10.2 9 Merger B.5 7.2 7.3 6.8 32 45 67 17 15.4 13.9 Hrazden MC Merger B.5 6.5 6.2 5.8 22 22 33 14.8 17.8 13.8 Ijevan MC Source: World Bank project documents; World Bank 2010. 74 Figure C.6. Trends of Hospital Discharges and Hospital Beds Sources: European Health for All database (HFA-DB). Updated June 2018. Armenian data from National Health Information Analytic Center, Ministry of Health of Armenia. Figure C.7. Average Length of Stay, All Hospitals (number of days) Sources: European Health for All database (HFA-DB). Updated June 2018. Armenian data from National Health Information Analytic Center, Ministry of Health of Armenia. 75 Figure C.8. Armenia and Comparators: Average Length of Stay, All Hospitals (number of days) Sources: European Health for All database (HFA-DB). Updated June 2018. Armenian data from National Health Information Analytic Center, Ministry of Health of Armenia. Figure C.9. Share of Public Health Expenditures by Provider, 2007–14 60% HP.1 Hospitals 50% HP.2 Nursing and residential care facilities 40% HP.3 Providers of ambulatory health care 30% HP.5 Provision and 20% administration of public health programs HP.6 General health 10% administration and insurance HP.8 Institutions providing 0% health-related services 2007 2008 2009 2010 2011 2012 2013 2014 Source: Armenia, Ministry of Health, 2015. Table C.13. Utilization Rate of Health Services by Poorest Quintile, 2010–15 (percent) Indicator Before Rationalization 2010 After Rationalization 2015 Utilization of outpatient health services 3.9 5.3 Utilization of inpatient health services 5 9.7 Source: World Bank 2016, 31; based on Armenia, Ministry of Health 2016. 76 Figure C.10. Ararat Medical Center, October 2018 a. Hospital b. Polyclinic 77 Figure C.11. Individuals Not Seeking Medical Care When Ill or Injured, 2012–14 (percent) Source: Armenia, Ministry of Health 2015. Figure C.12. Health Facility Responsiveness Domains at Yerevan PHC and Hospital Sites, 2012, 2016 Source: Armenia, Ministry of Health 2016. 78 Figure C.13. Health Facility Responsiveness Domains at PHC and Hospital Sites of Marz Cities, 2012 and 2016 Source: Armenia, Ministry of Health 2016. Table C.14. Number of Beds at Hospitals Benefited by APL II, Before and After Rationalization Number of Beds Marz Hospital Before rationalization After rationalization Ararat Ararat MC 80 50 Aragatcotn aparan MC 80 45 Armavir Armavir MC 185 110 Gegharkuniq Gavar MC 95 85 Chambarak HC 55 30 Kotayq Hrazdan MC 155 80 Abovyan MC 100 55 Shirak Gyumri MC 245 200 Lori Alaverdi MC 50 40 Syuniq Goris MC 180 90 Kapan MC 190 105 Meghri MC 45 45 Tavush Berd Mc 60 30 Ijevan MC 120 70 Totals 1,640 1,035 Source: World Bank project documents, World Bank 2016. Note: MC = Medical Center 79 Appendix C References Armenia, Ministry of Health. 2015. Armenia: Health System Performance Assessment. Yerevan, Armenia: Ministry of Health. Armenia, Ministry of Health. 2016. Armenia: Health System Performance Assessment. Yerevan, Armenia: Ministry of Health. 2002 Operations Policy and Country Services; World Bank. 2001. “World Bank Lending Instruments: Resources for Development Impac t.” Operations Policy and Country Services. Washington, DC: World Bank. ———. 2002. “Adaptable Lending: Third Review of Experience .” Operations Policy and Country Services. Washington, DC: World Bank. ———. 2004. “Armenia—Health System Modernization Project.” Project Appraisal Document 27708, World Bank, Washington, DC. ———. 2007. Environmental, Health, and Safety Guidelines for Health Care Facilities. World Bank, Washington, DC. ——— 2007b. “Armenia—Health System Modernization Project (APL II).” Project Appraisal Document 38149–AM, World Bank, Washington, DC. ——— 2010. “Armenia—Health System Modernization Project.” Implementation Completion and Results Report ICR1650, World Bank, Washington, DC. ———. 2012. Mainstreaming Environmental Management in the Health Care Sector: Implementation Experience in India and A Tool-kit for Managers. 2 vols. Washington, DC: World Bank. ——— 2016. “Armenia—Health System Modernization Project (APL II).” Implementation Completion and Results Report ICR3703, World Bank, Washington, DC. ———. 2017. “Multiphase Programmatic Approach.” Operations Policy and Country Services. Washington, DC: World Bank. 80 Appendix D. List of Persons Met WORLD BANK Name Title Mr. Alexan Hovhannisyan Senior Operations Officer, World Bank Mrs. Marianna Koshkakaryan Former Monitoring and Evaluation Specialist at HPIU (2013– 2015) GOVERNMENT Mr. Sergey Khachatryan Deputy Minister of Health, Ministry of Health Former Director of HPIU (1999–2010) Mr. Arsen Davtyan Deputy Minister of Health, Ministry of Health Former Head of Financial flows in SHA Mr. David Melik-Nubaryan Acting head of Health Care Policy Department, Ministry of Health Mrs. Hasmik Harutyunyan Head of Division Rapid Response for Complaints and Applications of Citizens at Ministry of Health Mr. Saro Tsaturyan Former head of the SHA Mrs. Gohar Panajyan Former Deputy Director of Health and Labour Inspection Body, Ministry of Health Mrs. Nune Bakunts Deputy Director National Center for Disease Control and Prevention Mrs. Sona Harutyunyan Former head of State Employment Agency Mr. Pavel Safaryan Former chief adviser to the Minister of Economic Development and Investments and former Deputy Minister of Finance; coordinator of project activities Mr. Alexander Bazarchyan Director of National Institute of Health Mrs. Diana Andreasyan Head of the Health Statistics department, National Institute of Health Mrs. Naira Davtyan Head of National Health Accounts, National Institute of Health Mr. Nelson Zuloyan 2012–2017 Former head of HPIU 2017–current–Vardanants Medical Center, Yerevan Mr. Yervand Elibekyan Head of primary health care component, HPIU Mrs. Christina Sargsyan Institutional development component coordinator, HPIU Mr. Hayk Sayadyan Head of HPIU Mrs. Diana Martirosova Former head of Household Survey Division of the National Statistical Service (NSS) from 2004–16 Mrs. Lusine Markosyan Head of Household Survey Division of the NSS (from 2001 at NSS) OTHER ORGANIZATIONS Mrs. Hasmik Harutyunyan Global Fund Program Coordination Team Manager Former head of hospital optimization component Mrs. Tatev Khachatryan Journalist at Hetq.am Mrs. Lusine Budaghyan Journalist at Aravot.AM 81 Mrs. Nvard Khachatryan Cofounder of Meghri Women’s resource center Pediatrician at Meghri medical center Trainer of nurses, doctors upon request from NIH Mrs. Astghik Grigoryan Project Management Specialist, Democracy, Health, and Social Reform Office, USAID Mr. Mikael Narimanov 2012–2016–Rector of the Yerevan State Medical University Mr. Robert Borisovich Avagyan 2003–2018–Deputy Director of Surb Grigor Lusavorich Medical Center Currently senior surgeon at Surb Grigor Lusavorich Medical Center (SGLMC) Mr. Manukyan Gagik 1st Deputy Director in Medicine and General affairs at Surb Grigor Lusavorich Medical Center (SGLMC) Mr. Manukyan Artak Deputy Director in line of Surgery and Research Methodology at Surb Grigor Lusavorich Medical Center (SGLMC) Mr. Manukyan Petros Deputy head of general medicine at Surb Grigor Lusavorich Medical Center (SGLMC) Mrs. Tadevosyan Nelli Head of Records Management and International Projects Department at Surb Grigor Lusavorich Medical Center (SGLMC) Mrs. Shaghbatyan Leyla Deputy Director in Finance at Surb Grigor Lusavorich Medical Center (SGLMC) Mrs. Mara Arsenovna Assistant to pro-rector on policlinic services of HUH Mr. Hambardzum Simonyan Deputy Country Director, Health Care Programs Director, Fund for Armenian Relief (FAR) Mr. Vahe Qrmoyan Health Care and Social Programs Officer Fund for Armenian Relief (FAR) Mr. Samvel Hovhannisyan Head of Armenian Association of Family Physicians Mrs. Diana Ter-Stepanyan Transparency International Anticorruption NGO Mrs. Donara Hakobyan Former deputy head of Yerevan State Basic Medical College (YSBMC) Mrs. Naira Nersisyan Deputy head of Yerevan State Basic Medical College (YSBMC) Mrs. Lusine Aydinyan International Consultant at UN 2004–2006 –Former Institutional Development component coordinator at HPIU Ms. Naira Gharakhanyan Director of Volunteer Health, Peace Corps Armenia 2010 –2015 leading the World Vision Health Advocacy and Policy programs and Child Health Now Campaign 2004–2010 leading health and social programs at Children of Armenia Fund (COAF) Mr. Aram Mnatsakanyan Head of Mkhchyan Rural Ambulatory Mrs. Naira Nersisyan Nurse at Mrgavet Health Post Mr. Arayik Sardaryan Head of the Ararat Medical Center 82