Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Report Number : ICRR0020399 1. Project Data Project ID Project Name P104467 HLTH SYS MOD (APL2) Country Practice Area(Lead) Additional Financing Armenia Health, Nutrition & Population P121728 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79870,IDA-42670 31-Dec-2012 30,420,000.00 Bank Approval Date Closing Date (Actual) 08-Mar-2007 29-Feb-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 22,000,000.00 0.00 Revised Commitment 40,955,943.78 0.00 Actual 41,858,566.52 0.00 Prepared by Reviewed by ICR Review Coordinator Group Judyth L. Twigg Robert Mark Lacey Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The project was the second phase of a two-phase Adaptable Program Lending (APL) program, the objectives of which were "to improve the organization of the health care system in order to provide more accessible, quality and sustainable health care services to the population, in particular to the most vulnerable groups, and to better manage public health threats" (Project Appraisal Document, Health System Modernization Project [APL1], May 2004, p. 5). The objectives of this project, according to the Financing Agreement (p. 5), were "to strengthen the Ministry of Health's (MOH's) capacity for more effective system governance, scaling up family medicine-based primary health care and upgrading selected healthcare service delivery networks in the Selected Marzes (regions) to provide more accessible, quality and sustainable health care services to the population." Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) At a December 2010 Additional Financing (AF) and restructuring, the objectives were revised to: "to strengthen the MOH's capacity for more effective system governance, scaling up family medicine-based primary health care and upgrading selected healthcare service delivery networks/infrastructure in the Selected Marzes to provide more accessible, quality and sustainable health care services to the population." Although additional activities (upgrading of infrastructure) were added at this restructuring, the core objectives of strengthening capacity for more effective system governance and providing more accessible, quality, and sustainable health care services did not change. ("Scaling up family medicine-based primary health care" and "upgrading selected healthcare service delivery networks/infrastructure in the Selected Marzes" are interpreted as means toward provision of more accessible, quality, and sustainable health care services.) However, some key outcome targets were revised downward, and therefore a split rating will be performed. At the time of this restructuring, US$ 18.7 million (44.9% of total Bank financing) had been disbursed. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 20-Dec-2010 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The project had four components: A. Family Medicine Development (appraisal US$ 4.7 million; AF added US$ 5.45 million [for a total estimate of US$ 10.15 million]; actual US$ 9.87 million) was to support strengthening of institutional capacity to train well-qualified family physicians and nurses as first-line primary health care (PHC) providers and improve their working environment. It was to complete planned training and re-training of 1,650 physicians and an equal number of nurses to ensure 100% population coverage, based on a ratio of one team per 1,700 - 2,000 population. About 50 rural ambulatories were to be upgraded, and outreach activities conducted to promote community participation. B. Hospital Network Optimization (appraisal US$ 20.77 million; AF added US$ 17. 0 million [for a total estimate of US$ 37.77 million]; actual US$ 43.23 million) was to support the implementation of optimization plans in eight Marzes that had not been covered by upgrading selected hospitals and refurbishing them with modern medical, information technology (IT), and health care waste management (HCWM) equipment. This component was also to finance technical work for architectural design, and training in hospital management, quality assurance, accountability and fiduciary management arrangements, and HCWM. Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) C. Institutional Strengthening (appraisal US$ 2.58 million; AF added US$ 2.24 million [for a total estimate of US$ 4.82 million]; actual US$ 0.66 million) was to strengthen MOH's capacity for policy making, planning, regulation, human resources development, and monitoring and evaluation, for more effective system governance and control of non-communicable diseases (NCDs). It was also to support strengthening the governance and management structures of health care facilities and the oversight function of Marz administrative structures. Support was to be made available to strengthen State Health Agency (SHA) operations, and to improve costing of publicly financed services and reimbursement mechanisms. The State Medical University (SMU) was to benefit from consultancy services to upgrade its medical curriculum, improve its teaching and training facilities, and introduce new technologies for continuous medical education (CME). D. Project Management (appraisal US$ 1.57 million; AF added US$ 0.63 million [for a total estimate of US$ 2.2 million]; actual US$ 1.79 million) was to provide institutional support to the MOH through a Health Project Implementation Unit (HPIU), which was to be in charge of implementing day-to-day project activities and monitoring and evaluation (M&E). The component was to finance annual financial audits as well as training and operating costs of the HPIU. At the 2010 AF, funds were added to support the rehabilitation of merged hospitals and the construction of one new hospital. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The initial estimated project cost was US$ 29.62 million. With the AF, estimated total costs were US$ 54.94 million. Actual total costs were US$ 55.59 million (with the difference due to exchange rate fluctuations). More was spent on health facility renovation than originally planned, and significantly less on institutional development and project management. The government decided to allocate public funds toward the latter activities, reducing the need for project financing. Financing: The project was originally financed by a US$ 22 million credit from the International Development Association (IDA). Additional Financing in the form of a US$ 19 million loan from the International Bank for Reconstruction and Development (IBRD) was approved in December 2010. Actual disbursements were US$ 22.71 million of the IDA credit and US$ 18.98 million of the IBRD loan, with the difference due to exchange rate fluctuations. Borrower Contribution: The Borrower and local communities initially committed US$ 7.62 million: US$ 7.17 from the government, US$ 0.3 million from the SMU, and US$ 0.15 million from local communities. An additional US$ 6.32 million was added at the AF, for a total planned Borrower commitment of US$ 13.94 million. The actual total contribution was US$ 13.89 million, with US$ 13.51 million from the Government and US$ 0.38 million from local communities. The planned contribution from the SMU was not made. Dates: • March 19, 2010: A level 2 restructuring added activities for rehabilitation of merged hospitals, construction of a new hospital, and provision of medical equipment. • December 20, 2010: A level 1 restructuring and AF added activities for new hospital investments, revised the results framework to include new outcome targets, and extended the project closing date from December 31, 2012 to December 31, 2014 to accommodate the new hospital investment plan. Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) •March 26, 2014: A level 2 restructuring extended the project closing date from December 31, 2014 to February 29, 2016 for completion of civil works in hospitals. The project closed three years and two months later than originally scheduled. 3. Relevance of Objectives & Design a. Relevance of Objectives The project's objectives were highly relevant to country conditions at appraisal. Armenia was on target to achieve most of its Millennium Development Goals, and it compared well, in terms of health outcomes, with other countries of similar socio-economic development. However, access to, and use of, health services remained low and was inefficient, favoring polyclinics and hospitals over PHC facilities. Health system governance was undergoing processes of change and reform, with the MOH taking early steps toward effective stewardship of the sector. Resources were poorly pooled and inequitably used. Efforts to reduce excess hospital capacity were under way but needed to be scaled up. The health workforce was still skewed toward overprovision with specialist care. The objectives to improve health system governance, access, quality, and sustainability of health care responded directly to these concerns. The objectives were also highly relevant to country and Bank strategy. The Bank's Country Assistance Strategy at entry (2005-2008) contained a pillar for reducing non-income poverty, advocating increased social sector spending and implementation of systemic social sector reforms. The Country Partnership Strategy at closure (2014-2017) has a strategic engagement cluster on improving efficiency and targeting of social/health services. Armenia's Development Strategy 2025 contains a pillar on enhancing human capital through improved access to quality social services (including health), and a separate pillar on improving social protection by enhancing efficiency of existing systems. The government health sector reform agenda continues to focus on strengthening PHC on the basis of family medicine, separating the health purchaser and provider functions by strengthening SHA capacity and reforming provider payment mechanisms, and completing optimization of the health services network (including hospital rationalization). The objectives did not change at the restructuring that revised key outcome targets. Relevance of objectives therefore remained High. Rating Revised Rating High High b. Relevance of Design Relevance of design was Substantial under both the original and revised outcome targets. There was a logical and plausible link between planned activities and expected outcomes. The technical assistance under the institutional strengthening component was appropriate for improved system governance. Planned training of physicians and nurses in family medicine, as well as physical upgrades to PHC centers and hospitals, were Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) directly linked to improved access to and quality of health care. Sustainability was to be achieved through optimization of the hospital network, which would be supported through upgrades to hospitals in parallel with closing of excess capacity and mergers where appropriate. The political sensitivity of this optimization process was recognized and was to be addressed through planned community outreach mechanisms. Provider payment mechanisms and other health financing reforms, also key to improving sustainability, were ongoing from previous projects and included costing exercises and support for improving the contracting function of the SHA. Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Strengthen the MOH's capacity for more effective system governance Rationale Outputs Technical assistance was provided to strengthen administrative, analytical, policy making, planning, regulatory, M&E, and hospital management capacity. Particular attention was also given to support strengthened purchasing of health care services and economic/cost analysis in the health sector. Outcomes A Health System Performance Assessment (HSPA) was published in 2008, 2010, and annually from 2012- 2015, exceeding the original and revised target of publication every two years. National Health Accounts reports have been produced annually since 2006, meeting the target of annual publication. Evidence gathered through these reports impacted a range of policy decisions. For example, data and analysis on increased unit costs helped determine the case reimbursement levels for services provided in regional hospitals; findings from a costing study led to a decision to introduce co-payments; overall cost data and analysis contributed to the introduction of global budgeting (operating on a pre-determined fixed budget for a specified period of time) in hospitals; and HSPA findings on the impact of screening programs at the primary care level led to increased investments in screening measures for hypertension, diabetes, and cervical cancer. The experience of producing the HSPAs and National Health Accounts reports, as well as their findings, contributed to evidence-based policy decisions and to more effective system governance. Planned Public Performance Reports for hospitals were not introduced as planned, due to cost factors and the lack of an appropriate regulatory framework. The percentage of SHA contracts concluded no later than 30 days after Parliamentary budget approval was 100% in 2016, meeting the target (the indicator was dropped at the AF because it had been achieved). The country's two hospitals with annual revenue exceeding AMD 1 billion published independent financial audits, as required by law. Five additional hospitals with revenue below that threshold also performed audits, not meeting the revised target of 11. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Overall, the technical assistance provided by the project increased MOH, SHA, and individual hospital capacity for data analysis, cost analysis and planning, policy making, contracting, and auditing, all contributing to more effective and responsive health system governance. Achievement of this objective is therefore rated Substantial. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective This objective (Strengthen the MOH's capacity for more effective system governance) was not revised, but outcome targets were revised at the 2010 AF. Revised Rationale As noted above: HSPA and National Health Accounts reports were published at a rate meeting or exceeding revised targets, with demonstrated impact on policy and governance. Smaller hospitals did not meet the revised target for performing audits. Overall, achievement of this objective under the revised targets is rated Substantial. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective Provide more accessible health care services to the population Rationale Outputs The project supported implementation of a one-year specialized training/retraining program in family medicine. The number of trained, certified family doctors increased from 633 in 2006 to 1,676 in 2016, exceeding the original target of 1,650 but not the revised target of 1,750 (due to a number of practicing doctors reaching or close to retirement age). The number of trained, certified family nurses increased from 568 in 2006 to 1,804 in 2016, exceeding the original target of 1,650 but not the revised target of 1,804. Including management professionals in hospitals, 4,118 total health personnel received training, exceeding the target of 3,700. 112 health facilities in eight regions were constructed, renovated, and/or equipped, exceeding the revised target of 100, including PHC facilities in 50 rural communities. Outcomes The number of people receiving services through the project-supported PHC network increased from 370,000 in 2006 to 627,000 in 2016, exceeding the revised target of 590,000. The number of people Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) receiving services through renovated hospitals reached 2.026 million, exceeding the revised target of 2 million. The number of annual admissions to project hospitals increased from 33,734 in 2010 to 49,121 in 2016, exceeding the revised target of 40,000. Overall, as of 2016, 99.68% of Armenia's population are enrolled in, and have access to, a PHC facility. The percentage of PHC-level professionals retrained and certified as family medicine practitioners increased from 47% in 2006 to 94.05% in 2016, not meeting the original target of 100%, but essentially meeting the revised target of 95%. A small number of doctors and nurses who had retired or were close to retirement age were not retrained. Per capita PHC visits increased from 2.0 in 2003 to 2.8 in 2007 and 4.0 in 2013 (capturing the impact of both APL1 and APL 2). The hospitalization rate similarly increased from 6.9% in 2003 to 8.4% in 2007 and 12.3% in 2013. Utilization of health services by the poorest income quintile increased from 3.9% in 2010 to 5.3% in 2014 (outpatient) and 5.0% to 9.7% (inpatient), meeting the revised target of increased utilization. The ICR (pp. 15-16) also demonstrates that increases in utilization of health services were more pronounced in disadvantaged than in relatively better-off regions of the country. The percentage of payments for essential health services that was out-of-pocket decreased from 35.4% in 2010 to 20.6% in 2014 for PHC, and from 47.1% to 42.8% for hospitals, meeting the original and revised target of any reduction. Based on these data on overall utilization and utilization by the poor at both the inpatient and outpatient levels, training of family doctors, and out-of-pocket spending data, achievement of this objective is rated Substantial under the original targets. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective This objective ( Provide more accessible health care services to the population) was not revised, but outcome targets were added or revised at the 2010 AF. Revised Rationale As noted above: Targets for the number of people receiving services through project-supported PHC facilities or renovated hospitals were added at the AF and were reached or exceeded. The revised target for retraining and certifying of PHC-level professionals as family medicine practitioners was essentially met. Utilization of both inpatient and outpatient health services by the poorest income quintile increased, meeting the revised target of any increase. Out-of-pocket health care costs decreased, meeting the revised target of any decrease. Based on these data, achievement of this objective under the revised targets is rated Substantial. Revised Rating Substantial PHEFFICACYTBL Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Objective 3 Objective Provide more quality health care services to the population Rationale Outputs In addition to the outputs listed above, the project provided assistance with quality assurance mechanisms at the PHC and hospital levels. Based on project technical support, the SMU revised its curriculum, teaching methodologies, and student evaluation system to align with European Union standards. Continuing medical education was supported at Yerevan State Medical University. Outcomes The percentage of patients with hypertension who had at least one electrocardiogram annually increased from 42% in 2010 to 55.2% in 2014. The percentage of patients with ischemic heart disease who had at least one total cholesterol test annually increased from 33% in 2010 to 53.7% in 2014. Based on Health System Performance Assessment surveys, positive perceptions of quality of care increased from 64.3% in 2007 to 73.2% (PHC rural), 73.7% (urban hospital), and 66.8% (rural hospital) in 2012, and remained essentially the same in urban PHC facilities (64.1%), meeting the original target of some increase and approaching the revised target of 75% (there is a single baseline because the 2007 survey results do not differentiate by type of facility). The ICR further notes that utilization rates increased in district-level hospitals at a faster pace than that for the capital, Yerevan, a possible indication that people were choosing to receive care locally as project-supported improvements increased perceived quality of care at the local facilities. This point is supported by a decrease since 2012 in the absolute number of hospital admissions in the capital and in private sector hospitals since 2012, but an increase in admissions to project-renovated district-level hospitals (ICR, pp. 17-18). Based on service and survey data, achievement of this objective under the original targets is rated Substantial. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 3 Revision 1 Revised Objective This objective (Provide more quality health care services to the population) was not revised, but a numerical outcome target was specified at the 2010 AF. Revised Rationale As noted above: Positive perceptions of quality of care essentially met the revised target of 75% in rural primary care facilities and urban hospitals. The ICR presents further evidence based on utilization data (described above) suggesting quality improvements in project-supported district-level hospitals. On this basis, achievement of this objective under the revised target is rated Substantial. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Revised Rating Substantial PHEFFICACYTBL Objective 4 Objective Provide more sustainable health care services to the population Rationale Outputs In addition to outputs reported above: The project supported improvements in preventive health services (hypertension, cholesterol testing, etc., noted above), intended to reduce the burden of relatively expensive care for late diagnosis with chronic disease. The project completed a process of consolidating 24 existing hospitals into ten hospital networks, achieved according to a hospital master plan created by the government with project support. All project hospitals applied updated environmental management guidelines. Outcomes The number of beds in 14 hospitals supported by project interventions decreased from 1,640 to 1,035. The number of existing square meters of hospital space in all regions decreased by 85% of planned reductions, meeting the original target of any reduction and the revised target of 85% (the ICR does not provide the exact figure for space reduction). The average length of stay in regional hospitals decreased from 7.7 days in 2006 to 5.8 days in 2016, surpassing the original target of any decrease and the revised target of 6.5 days. Hospitalization rates increased, as reported above, reflecting increased usage at the smaller number of hospitals in service after mergers/consolidations.c The proportion of the health budget allocated to PHC increased from 36.4% in 2006 to 37.2% in 2016, not reaching the original target of 45% or the revised target of 43%. The ICR (p. v) reports that public expenditure on health increased from AMD 31 billion in 2005 to AMD 76.6 billion in 2015, even in the aftermath of the 2009 financial crisis; however, inflation rates varied from around zero to over 11% during this time period, and so much of this increase was mitigated by inflation. The project team later added that about half the increase was due to inflation, and the other half represented real gains. In the post-socialist context, sustainability refers to financial sustainability accrued through reducing over- reliance on expensive hospital networks and outpatient specialists, and increasing the capacity and utilization of more inexpensive and efficient primary and preventive care (with a focus on family medicine). While it is too soon to demonstrate reduced costs per intervention or per capita costs, it is likely that the rationalization achieved under the project has contributed to sustainability of the system through increased efficiency of service delivery (more and better health services delivered within the same budget envelope). Importantly, project-supported reforms have improved health services utilization and reduced out-of-pocket payments while health spending as a percentage of GDP remained essentially constant (1.39% in 2005, 1.58% in 2014), even through the 2009 financial crisis and its aftermath. Achievement of this objective under the original targets is therefore rated Substantial. Rating Substantial Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) PHREVDELTBL PHINNERREVISEDTBL Objective 4 Revision 1 Revised Objective This objective (Provide more sustainable health care services to the population) was not revised, but outcome targets were added or revised at the 2010 AF. Revised Rationale As noted above: The project met the revised target for decrease in hospital space reduction, and significantly exceeded the revised target for reduction in average length of hospital stay. The revised target for increased budget allocation to primary care, however, was not met. Overall, achievement of this objective under the revised targets is rated Substantial. Revised Rating Substantial PHEFFICACYTBL Objective 5 Objective Program objective: improve the organization of the health care system in order to provide more accessible, quality and sustainable health care services to the population, in particular to the most vulnerable groups, and to better manage public health threats Rationale The project's outcomes represent progress toward achievement of the overall Program objectives, including attention to the most vulnerable groups (as measured by access indicators for the lowest income quintile, as well as hospital utilization patterns for outlying regions in comparison to the capital). The ICR also reports improvement in maternal mortality, from 35/100,000 live births in 2006 to 19/100,000 in 2013. although this outcome is also attributable to many other factors outside the scope of the Program. Rating Not Rated/Not Applicable PHREVDELTBL PHREVISEDTBL 5. Efficiency The Project Appraisal Document's economic and financial analysis (pp. 81-87) focused on sector efficiency and likely future commitments to public spending on health. The ICR's discussion is similar, without a traditional cost-benefit or rate-of-return analysis. The ICR's discussion (pp. 30-33) also focuses on gains in sector efficiency rather than project efficiency. Qualitatively, the project's investments are known "best buys" for contributing to access (primary care Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) strengthening), quality (facility renovation, equipment, and training), and sustainability (in the post-socialist environment, hospital streamlining to reduce dependence on relatively expensive inpatient care). In an environment where resources had historically been skewed toward the hospital sector at the expense of primary care, overall access to care was most efficiently addressed through conversion of existing specialists and training of new medical graduates in the primary care field, as well as construction and upgrading of rural primary care facilities in areas that had been previously underserved. These relatively inexpensive primary care training programs also made a cost-effective contribution to improvement in quality of services delivered. In comparison with other possible options for financial sustainability (reliance on external financing, price increases, or reductions in services), the structural transformation achieved through downsizing and consolidation of excess hospital capacity was the most efficient way to free resources to sustain (and even improve) overall service delivery without raising costs. Implementation efficiency was clearly strong, with an effective HPIU carrying out financial management and procurement with very few shortcomings. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The objectives under both the original and revised outcome targets were highly relevant to country conditions, government strategy, and Bank strategy. Relevance of design was substantial under both the original and revised targets, as planned activities were logically and plausibly connected to expected outcomes. Achievement of each of the project's four objectives -- improved governance, access, quality, and sustainability of health care -- was substantial under both sets of targets. Efficiency was substantial due to the selection of "best-buy" strategies for improving access, quality, and sustainability in the Armenian context, as well as strong evidence of cost-effective implementation. Taken together, these ratings are indicative of minor shortcomings in the project's preparation and implementation under both the original and revised outcome targets, and therefore an overall Outcome rating of Satisfactory. a. Outcome Rating Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Satisfactory 7. Rationale for Risk to Development Outcome Rating A new, US$ 35 million project is currently under implementation: the Disease Prevention and Control Project, focusing on improvement of maternal and child health services, prevention and management of selected non- communicable diseases, and enhancement of the efficiency and quality of selected hospitals. The government remains committed to sustaining the gains achieved under APL1 and APL2, and there is little political risk to the reforms introduced under the project, including the hospital streamlining efforts that have been so politically controversial in many other post-socialist countries. The Borrower's ICR (p. 38) stresses the likelihood of sustained or even increased budget allocations to the health sector, further reform of medical education to stress the training of primary care staff, and the institutionalization of key activities (HSPA and National Health Accounts reporting) that have contributed to more effective sector governance. a. Risk to Development Outcome Rating Negligible 8. Assessment of Bank Performance a. Quality-at-Entry At preparation, triggers for progression from APL1 to APL2 had been partially or fully met, and the government had already established a track record of implementing hospital mergers effectively in Yerevan. Design took lessons from APL1 and from a review of health reforms in transition countries: the need to enhance allocative efficiency by reorganizing access to PHC and introducing a gatekeeper function for access and referral to hospitals; the need for strategic investments in human resources; the extent to which rehabilitation of health facilities was key to raising the quality of health services; and the need to strengthen management capacity among providers and government. A full range of stakeholders was involved with project preparation, and there were strong synergies with a concurrent development policy loan. As was common in health sector reform in transitional economies at the time, the United States Agency for International Development was a key partner, financing in parallel the enhancement of PHC practices and promoting their acceptance through public information and education campaigns. Risk assessment was thorough (with highest risk assigned to the possibility that government would be reluctant to pursue politically sensitive hospital rationalization plans) and mitigation measures well conceived. Institutional and implementation arrangements were sound, with MOH responsible for M&E in close coordination with the Health Information and Analytic Center of the National Institute of Health (carrying over arrangements from the APL1). Appropriate arrangements for ensuring safeguards and fiduciary compliance were also held over from the first phase of the APL. M&E design was adequate, though some key outcome indicators lacked baselines and targets (see Section 10a). Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Quality-at-Entry Rating Satisfactory b. Quality of supervision According to implementation status reports (ISRs), implementation support missions were appropriately timed and staffed, and the project team included members with extensive experience in the region and in hospital rationalization. The ICR (p. 22, citing evidence from internal communications, letters to the government, and Aides Memoires) notes that project restructurings addressed issues in a timely manner, and that communication between the Bank team and the government was diligent and effective, including sound policy advice, the production of a Public Expenditure Review for the health sector, and policy workshops. The ICR team observed strong and cordial relationships between the Bank team and government offices (ICR, p. 22). M&E and safeguards supervision was adequate (see Sections 10b and 11a), including the setting of initially-missing baselines and targets for some key outcome indicators. ISR reporting appears to have been thorough and candid, with ratings downgraded temporarily to Moderately Satisfactory when shortcomings arose. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. Assessment of Borrower Performance a. Government Performance The government, including the MOH, had a solid and long-term commitment to health sector reform, as demonstrated by its support for development of primary care beginning in 1997 and for hospital streamlining from the first phase of the APL. Although hospital rationalization, involving mergers and consolidations, is politically sensitive, the government enacted a decree to restructure 24 hospitals into ten networks; it also prepared and approved a master plan for optimization of remaining hospitals. A Steering Committee composed of representatives of key stakeholders, both within and external to the MOH, provided overall supervision and oversight for the project, maintaining a transparent and participatory approach (ICR, p. 8). Government co-financing for the project was always timely and complete, despite pressures from the 2008-2009 financial crisis. MOH institutionalized the production of key reports now central to policy formulation. Government Performance Rating Satisfactory Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) b. Implementing Agency Performance The HPIU, which had implemented APL1 and the human health component of an Avian Influenza Preparedness Project, had adequate financial management and procurement arrangements in place. It coordinated and offered technical assistance to participating institutions. Corruption was perceived as an important issue, and mitigation measures -- a formal internal control framework, flow of funds mechanism, independent auditing of financial statements, and regular financial management and procurement supervision and review -- were put in place. Over the course of the operation, the HPIU was fully compliant with safeguards and fiduciary requirements and was "highly effective" in overseeing day-to-day project activities (ICR, p. 8). Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design MOH and the Health Information and Analytic Center were responsible for monitoring project progress and outcomes. Project indicators were to be tracked through routine administrative data (health status and health care utilization), surveys performed on an ongoing basis by the National Statistical Service, and additional surveys of health care users and providers. Output and outcome indicators were complete and adequate measures of achievement, but many key outcome indicators lacked baselines and targets. b. M&E Implementation All indicators were systematically monitored and reported on, as planned. At the 2010 restructuring, baselines that had been missing at appraisal were established, some indicators were dropped due to measurement challenges, and others were revised or targets were revised. The Health Information and Analytic Center regularly produced Health System Performance Assessments and National Health Accounts reports. Special attention was paid in these reports to equity issues, including trends in informal payments and the impact of health programs and policies on the poor. c. M&E Utilization The project's M&E data and analyses were used to inform policy decisions, including hospital rationalization plans and the project's own additional financing. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was rated Environmental Assessment Category B. Other than Environmental Assessment (OP/BP 4.01), no safeguard policy was triggered. The PAD reports that the Environmental Management Plan (EMP) from the first phase of the APL remained in place, governing site-specific screening for all project-supported rehabilitation of PHC centers and hospitals, and that an environmental management framework was prepared and publicly disclosed in December, 2006. Health care waste management guidelines and training were put in place. According to the ICR (p. 9), compliance with the EMP and with the Bank's safeguard policies was satisfactory. b. Fiduciary Compliance Financial management arrangements under the project, including accounting, reporting, budgeting, internal control, funds flow, and staffing, were rated fully Satisfactory in supervision reports during the project's entire duration with only one exception (the last financial management mission in November 2015, when deficiencies in internal control, related to delays in recovery of an advance to a building contractor, led to a Moderately Satisfactory rating) (ICR, p. 10). Quarterly interim financial reports were always received from the HPIU on time and found acceptable. Annual financial audits were on time and unqualified. The Procurement Unit of the HPIU proceeded in line with Bank guidelines, and procurement was rated Satisfactory throughout the project's lifetime (ICR, p. 10). c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Risk to Development Negligible Negligible --- Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) Outcome Bank Performance Satisfactory Satisfactory --- Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR (pp. 23-25) provides insightful lessons, some of which are reformulated here: Political commitment is the key prerequisite to health sector reform in the post-socialist context. In this case, government support at the highest levels for politically sensitive hospital rationalization remained consistent, enabling gains in sector efficiency through consolidation of hospitals and prioritization of primary care. The APL is a valuable instrument in situations of long-term commitment to reform. In Armenia, reform of the health sector has spanned three Bank projects and two decades. This instrument enabled the Bank to sequence activities and refine targets as progress was made toward long-term goals. Infrastructure is not enough. For health system modernization, it is important to invest in modernization of hospitals and clinics, but training, capacity building, and modernization of rules for provider payment and other key resource flows are also essential. Accurate political economy analysis is a prerequisite for health sector reform. In this case, there was potential for significant stakeholder opposition to hospital rationalization, but the government and Bank teams anticipated that opposition and effectively managed expectations and interests. 14. Assessment Recommended? Yes Please explain To demonstrate the value of long-term commitment in a sector, as well as the importance of solid political economy analysis to underpin potentially controversial reforms. 15. Comments on Quality of ICR Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review HLTH SYS MOD (APL2) (P104467) The ICR presents the project's background, implementation experience, and achievements with clarity, candor, and concision. It assiduously follows IEG/OPCS guidelines. Where appropriate, it brings to bear data from outside the project's formal results framework to assess achievement of objectives. It effectively places the project in the context of Armenia's overall health sector reform effort, and of the preceding APL1. Its lessons are comprehensive and well derived from the project's preparation and implementation experience. More information could have been provided on specific project activities (Annex 2 is rather thin). The ICR's discussion of efficiency would have been enhanced by more quantitative analysis. a. Quality of ICR Rating Substantial Page 17 of 17