Document of The World Bank FOR OFFICIAL USE ONLY Report No: 29233-AM IMPLEMENTATION COMPLETION REPORT (IDA-29790) ON A CREDIT IN THE AMOUNT OF US$10.0 MILLION TO THE REPUBLIC OF ARMENIA FOR A HEALTH FINANCING AND PRIMARY HEALTH CARE DEVELOPMENT PROJECT JUNE 23, 2004 Human Development Sector Unit Europe and Central Asia Region This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective June 4, 2004) Currency Unit = Dram (AMD) AMD548.40 = US$ US$ 1 = SDR1 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS BBP Basic Benefit Package BMC Basic Medical College CAS Country Assistance Strategy CDC Center for Disease Control CIS Commonwealth of Independent States DO Development Objective ECA Europe and Central Asia FM Family Medicine GDP Gross Domestic Product GOA Government of Armenia HIS Health Information System ICR Implementation Completion Report IDA International Development Association IMR Infant Mortality Rate Marz An administrative region in Armenia MMR Maternity Mortality Rate MOFE Ministry of Finance and Economy MOH Ministry of Health MTEF Mid-Term Expenditure Framework NGO Non-Governmental Organization NIH National Institute of Health NIS Newly Independent State PHC Primary Health Care PCU Project Coordination Unit PRSP Poverty Reduction Strategy Paper PSR Project Status Report SAC Structural Adjustment Credit SHA State Health Agency SMU State Medical University USAID United States Agency for International Development WHO World Health Organization Vice President: Shigeo Katsu Country Director Donna Dowsett-Coirolo Sector Manager Armin Fidler Task Team Leader: Toomas Palu ARMENIA Health Financing and Primary Health Care Development CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings 1 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 6 5. Major Factors Affecting Implementation and Outcome 16 6. Sustainability 18 7. Bank and Borrower Performance 19 8. Lessons Learned 21 9. Partner Comments 22 10. Additional Information 24 Annex 1. Key Performance Indicators/Log Frame Matrix 25 Annex 2. Project Costs and Financing 27 Annex 3. Economic Costs and Benefits 29 Annex 4. Bank Inputs 30 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 32 Annex 6. Ratings of Bank and Borrower Performance 33 Annex 7. List of Supporting Documents 34 Annex 8. Beneficiary Survey Results 35 Annex 9. Stakeholder Workshop Results 38 Appendix 1: List of Participants at Stakeholder Workshop 41 Project ID: P050140 Project Name: Health Financing and Primary Health Care Development Project Team Leader: Toomas Palu TL Unit: ECSHD ICR Type: Intensive Learning Model (ILM) of ICR Report Date: June 23, 2004 1. Project Data Name: Health Financing and Primary Health Care L/C/TF Number: IDA-29790 Development Project Country/Department: ARMENIA Region: Europe and Central Asia Region Sector/subsector: Health (68%); Central government administration (16%); Compulsory health finance (16%) Theme: Health system performance (P); Participation and civic engagement (S); Other communicable diseases (S); Injuries and non-communicable diseases (S); Rural services and infrastructure (S) KEY DATES Original Revised/Actual PCD: 10/06/1996 Effective: 03/31/1998 03/02/1998 Appraisal: 04/22/1997 MTR: 03/19/2001 03/19/2001 Approval: 07/29/1997 Closing: 06/30/2002 12/31/2003 Borrower/Implementing Agency: REPUBLIC OF ARMENIA/MINISTRY OF HEALTH Other Partners: STAFF Current At Appraisal Vice President: Shigeo Katsu Johannes Linn Country Director: Donna Dowsett-Coirolo Basil G. Kavalsky Sector Manager: Armin Fidler Thomas Blinkhorn Team Leader at ICR: Toomas Palu Alexandre Marc ICR Primary Author: Monika Huppi; Silviu Calin Radulescu 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: SU Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The project aimed to support implementation of the Government's health care reform program in the areas of primary care and health finance. Specifically, its objectives were to: (i) improve quality and efficiency of primary health care, through training and retraining of primary health care staff, introduction of practice guidelines and improving the infrastructure and equipment in selected PHC facilities; (ii) improve efficiency, transparency and targeting of public health spending by introducing a basic benefits package, performance based provider payment methods and modern financial management; and (iii) mobilize communities to take an active role in defining local health care priorities and sustaining basic health care services. The objectives were highly relevant to improving the performance of the health care system in Armenia and in line with one of the core objectives of the 1997 Country Assistance Strategy, which focused on ensuring social sustainability of reforms through strengthening of the social safety net and improvements in quality of and access to basic health and education services. The objectives continue to remain at the forefront today, as confirmed in the Government's recently completed medium-term health sector strategy, and evidenced by the fact that IDA's recently approved Health System Modernization Project continues to support these objectives, albeit under a broader perspective as appropriate for a second intervention in the sector. 3.2 Revised Objective: The objectives remained valid throughout project implementation and beyond and were thus not revised. 3.3 Original Components: Component 1: Strengthening Primary Health Care (US$6.1 million base cost). This component was designed to support the Government's Primary Health Care (PHC) Strategy, aiming to improve the quality and accessibility of PHC services in Armenia. Support was to be provided to develop training and re-training capacity of PHC providers; to rehabilitate and equip PHC facilities and train their staff in selected communities; and to develop PHC practice guidelines. The component had four sub-components as follows: Sub-Component 1.1: Primary Health Care Providers Training Program (US$2.2 million base cost). This subcomponent was designed to support conversion of primary health care level providers into family medicine practitioners, with more independence in decision making, able to offer comprehensive preventive and curative services in the outpatient setting and supported by qualified middle level personnel. The sub-component was to support retraining of existing primary health care providers (district therapists, district pediatricians, nurses and midwives) at the National Institute of Health and Medical College as well as training of undergraduates and new family practitioners at the State Medical University. It was estimated that 700 providers would be retrained. The project was to finance renovation and equipment for training, technical assistance for curriculum development, fellowships, short-term external training and workshops for training trainers. Sub-Component 1.2: Primary Health Care Development Program (US$3.5 million base cost). This subcomponent was designed to support improvement of the quality of PHC services in selected localities by involving communities in the management and financing, providing incentives to staff to improve services, - 2 - rehabilitating infrastructure and providing equipment and training. It was envisaged to cover 70 facilities based on applications by communities for PHC development grants that were prepared according to pre-set guidelines and including a commitment by communities to contribute 10 percent of project costs. The project was to finance civil works, medical equipment and furniture, pharmaceutical supplies, training for health providers, information campaign, transportation of equipment and supplies to facilities. Sub-Component 1.3: Primary Health Care Guidelines Development (US$0.4 million base cost). This subcomponent was designed to support development and dissemination of guidelines for PHC, based on international experience and adapted by local experts. The project was to finance refurbishment of office space for the working group, office equipment, study tours, foreign and local technical assistance and costs of printing and dissemination of guidelines. Component 2: Strengthening Health Financing System (US$3.6 million base costs). This component was designed to support the Government's health finance reform, which aimed at shifting from input based financing towards targeted and performance-based payments of health care providers. Key aspects of the reforms which the project aimed at supporting included: (i) separation of health care provision from health care financing; (ii) limiting Government funding for health care to a basic benefits package; and (iii) introducing improved provider payment methods aimed at improving efficiency in service provision. To this effect, the component had four subcomponent as follows: Sub-Component 2.1: Establishing a State Health Agency: (US$2.1 million base costs). This sub-component was designed to support the establishment the State Health Agency (SHA), as a quasi-autonomous public body which would contract health service providers for the provision of health services defined in the basic benefits package, and make decisions on the regional allocation of public funding for health care provision. Establishment of SHA was key to separating health care provision from financing. The project component was to support SHA establishment through technical assistance, training, renovation of premises to house SHA's headquarters, as well as its regional branch offices, provision of furniture, office equipment, vehicles and computer hard ware and software for its management information system. Sub-Component 2.2: Improvement of Basic Benefits Package Methodology (US$0.4 million base costs). This component was designed to provide local and international technical assistance and training to help define the contents of the basic health care benefits package (BBP) which would be made available to the population from public resources. The objective was for a technical working group to come up with a BBP which would be based on burden of disease and cost effectiveness analysis, address the most pressing health needs for the population and fit within the available resource envelope. In addition to technical assistance, the project also financed refurbishment of office space and equipment for the working group. Sub-Component 2.3: Improvement of Provider Payment Methodology (US$0.4 million base costs). The component was to finance international and local technical assistance and training in support of a working group that was to develop a methodology for cost-accounting and pricing, define and help introduce improved provider payment mechanisms, design standard formats for SHA ­ provider contracts, develop performance monitoring capacity and a strategy to legalize and regulate private payments to health care providers. The idea was to assure development of adequate methodologies to support the introduction of output based provider payments as SHA was being established. - 3 - Sub-Component 2.4: Financial Information System (US$0.7 million base costs). This component was to finance hardware, software and technical assistance, to develop a financial information system for SHA that would allow for efficient financial management of the provider payment process, including design of medical and financial reporting forms, automation of data collection and analysis and automation of agency payments to providers and accounting. Component 3: Project Management (US$1.0 million base costs). This component provided for a Project Coordination Unit (PCU) to assure overall project management and coordination. Specifically, the PCU was to plan and coordinate implementation of project activities; ensure that resources were utilized in accordance with implementation plans; manage procurement, contracting and disbursements; maintain financial accounts and records; monitor project progress and impact and report to the project management committee, the Minister and the Bank. The PCU was also coordinating and supporting the work of technical working groups which were established under most project sub-components. The PCU was staffed by a combination of project management and technical staff. 3.4 Revised Components: Component 1: There was no substantial restructuring of the primary care component during project implementation. However, the project faced difficulties in the beginning with mobilizing the community contribution at 10 percent of civil works cost. This was partly because of economic difficulties in the aftermath of the Russia financial and economic crisis in 1998 and also because the project targeted the poorest communities. An agreement was reached with the Bank to reduce the community contribution to 2 percent. Given the Dutch grant support to the health financing component of the project that materialized during the project implementation that allowed to reallocate credit resources, additional activities were added during implementation as follows: (i) establish and equip diagnostic departments at the marz (marz - administrative region in Armenia) central polyclinics; (ii) provide rehabilitated and newly constructed ambulatories and regional policlinics with cars; and (iii) equip the marz epidemiological stations with bacteriological laboratories. Component 2: None of the health finance sub-components were substantially restructured during project implementation and the MTR confirmed that both the relevant development objectives and the component design remained most pertinent to Armenia's health finance reforms. However, as Dutch grant financing became available to provide foreign technical assistance in support of the Government's health finance reforms, a decision was taken that IDA credit funds would only provide for local technical assistance (financing of working groups), local training, civil works and goods under the second project component. Funds which had originally been allocated towards technical assistance and long term fellowships abroad under the second project component were reallocated towards the first project component. In addition, the scope of the financial information system sub-component was expanded to provide support with the development of a broader health information system which would enable integration of health financing information with data on epidemiology and health system resources. - 4 - Component 3: The component was not revised during implementation. 3.5 Quality at Entry: The quality at entry was satisfactory. The project was adequately designed to meet the Government's reform priorities and match the implementation capacity of implementing agencies. The project's selectivity, focusing on two key aspects of the Government's health sector reform, rather than on a wide array of reforms and activities, was an important factor that contributed to successful implementation. The project was thoroughly prepared over a period of two years, during which the Bank was engaged in a continuous dialogue with the Government on sectoral policy reforms. While the Government had already decided on the broad reform directions at the time project preparation was initiated, project preparation contributed significantly to more detailed policy formulation and implementation planning of the reforms on the primary health care and health finance fronts. Project preparation was supported by core analytical work which significantly contributed to opertionalizing the policy reform agenda in primary health care and health finance and the conclusions of the analytical work were used to guide project design. The project was consistent with the objectives of the 1997 Country Assistant Strategy, which focused on consolidating recovery of economic growth through private sector development, and ensuring social sustainability of reforms through a strengthening of the social safety net and improvements in quality of and access to basic health and education services. The borrower was fully engaged in project preparation through two working groups at the Ministry of Health, with many of the working group members continuing the work through much of project implementation. Consultations with key stakeholders at all levels were held to discuss the project concept and assure broad support of the concept. While the overall project concept was well prepared and enjoyed strong ownership from the borrower, the extent to which implementation of fundamental changes in service delivery and health financing required time and effort for building institutional capacity, implementing and enforcing new regulatory and contractual arrangements, overcoming resistance by vested interests, and raising the awareness and buy-in by the population and politicians may have been somewhat under estimated. This resulted in somewhat unrealistic expectations as to what is achievable in a four year time frame in terms of number of primary care providers trained (appraisal estimate: 700; actual 159 doctors, 152 nurses) and the possible impact of health finance reforms on improved sectoral efficiency (e.g. appraisal target for hospital occupancy rate: 65%, actual in 2003 37%). Please also refer to section 4.2 for a more detailed discussion on outcomes and outputs. One design shortcoming was the fact that the health finance reform component focused only on the purchaser side of the purchaser-provider split, ignoring the need to establish appropriate mechanisms to ensure adequate accountability on the provider side, as well as the need to support reforms of financing mechanisms and reorganize and rationalize the excessive provider network. The importance of these two issues was recognized during the course of implementation and subsequently supported through hospital network optimization and public hospital financial management improvement programs under the structural adjustment lending operations (SAC III ­ SAC V). They also constitute a major focus of the recently approved Health System Modernization Project. These qualifications not withstanding, project design and relevance are judged to have been satisfactory as they focused on key aspects of the Government's reform program which remain valid even today, were in line with CAS objectives at the time and remain in line with the current CAS and the borrower was fully engaged during project preparation. - 5 - 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The project has broadly met its development objectives, as attested by achievement of most outcome indicators and general agreement among major stakeholders that the project has made a substantial contribution towards both the introduction of an improved family medicine based primary health care system and albeit to a lesser extent a new system of health care financing. The project has laid the foundations for a significantly improved family medicine based primary health care system which serves the population in target areas better than was previously the case and upon which the Government plans to build to assure provision of improved primary care country wide within the next six years. Communities became involved in decisions about basic services and contributed to the financing of PHC development projects in their localities. The project has also supported the introduction of fundamental changes in the health finance system, although the efficiency and targeting of public health spending still requires further improvements. The health finance system has overall suffered from serious under-funding throughout the entire project period which has limited the impact of reforms supported by the project (a factor which clearly was outside the control of the project) and additional attention should have been paid to strengthening the provider accountability framework. Objective 1: Improve quality and efficiency of primary health care through training and retraining of primary health care staff, introduction of practice guidelines and improving the infrastructure and equipment in selected PHC facilities. DO rating: Highly Satisfactory The project has contributed to the successful implementation of the PHC strategy, approved by the Government of Armenia in 1997. Project outcomes in areas supported by the project demonstrate that PHC is of better quality, more accessible and covering a broader range of population needs. Capacity for training family physicians and nurses has been firmly established at the Family Medicine chairs of Yerevan State Medical University, National Institute of Health and the Yerevan Basic Medical College. Evaluation of PHC services in the project intervention areas, that covered by the end of the project about 19 percent of the population, has documented the changes of accessibility, efficiency and quality. Access to primary health care has improved, as demonstrated by higher utilization rates when experiencing health problems (61 percent vs 53.7 percent) and higher use of local PHC services as first contact of care. Patients seeking care in communities served by a retrained family physician are only half as likely to pay for consultations as those in other communities. Total out of pocket payments for treatment in a family medicine setting were about 10 percent lower than in other primary care settings. Evidence of increased efficiency and reduced costs for patients is provided by the marked reduction of referrals and self-referrals for specialist and hospital care, as well as by improved accuracy of diagnoses at the PHC level. Patients trust PHC providers more in intervention areas, they receive greater advice regarding healthy behaviors and perceive quality of care as being high. The effectiveness of established retraining of family physicians and nursing programs in increasing participants' knowledge, skills and ability to provide a broader range of services was demonstrated by objective assessments performed before and after participation in the training programs. The external peer review of the PHC training program by a World Organization of Family Doctors (WONCA) expert confirmed the quality of the content and the good capacity established for delivery of the training in the three Armenian institutions supported by the project. The WONCA review also provided constructive recommendations for further improvement, such as to extend the duration of training, expand patient contact and access to practical skills development after establishing the clinical training bases in the - 6 - regions. There was a high uptake and utilization of clinical guidelines by newly trained health professionals, as well as all other primary health care staff. The guidelines proved to be effective tools for improving management of common conditions in primary care. There is demand for broader dissemination of the guidelines and for further development and updating. During the project life, the impact indicators monitored nationally (immunization rates, incidence of vaccine preventable diseases) have improved or remained at same level compared to baseline. This can be attributed in part to improved PHC services and is a good result in the context of severe financial constraints experienced by the health sector during the project. Objective 2: Improve efficiency, transparency and targeting of public health spending by introducing a basic benefits package, performance based provider payment methods and modern financial management. DO rating: Moderately Satisfactory. The achievements of this objective must be evaluated in light of the overall difficult fiscal situation and the precariously low public sector funding for health care which afflicted Armenia during most of the project implementation years. As a result of sharp decline in output (real GDP in 2000 was only two thirds of GDP in 1990) and the associated fiscal retrenchment during the early and mid-1990s, health sector spending in Armenia had dropped by close to forty percent during the first half of the 1990s. At the outset of the project, public sector spending on health care amounted to just one percent of GDP, although the Government made a commitment to accord increasing importance to social sector spending at the time. However, the effects of the 1998 Russia crisis and a relatively high debt service burden called for further fiscal adjustments during 1999-2001 and the Government largely failed to protect social sector spending during this time. As a result, health sector spending dropped to under one percent of GDP by 2000, while actual sectoral financing amounted to less than half the approved budget during that year (see table below). Efforts were made to increase budgetary allocations and actual budget execution to the health sector as of 2002 (when all arrears to providers were cleared and partly written off). However, overall spending on health care in Armenia remains among the lowest in the ECA region and has fallen significantly short of expectations set at appraisal. In 2003, Armenia public sector expenditures on health amounted to only 1.3 percent of GDP or 6.8 percent of total public sector spending, compared to an EU average of 6 to 8 percent of GDP and 13 percent of total public sector spending. At project appraisal, it had been foreseen that public sector spending on health care would increase to the regional average of about 3 percent of GDP. The severe sectoral underfunding and poor budgetary execution during most of the project period severely limited the extent to which project supported health finance reforms could have their full expected impact. - 7 - 1998 1999 2000 2001 2002 2003 GDP growth (%) 7.2% 3.3% 6.0% 9.6% 12.9% 13.9% Budget Deficit/GDP (%) -3.8% -5.2% -4.9% -4.3% -2.6% -1.3% BudgetaryExpenditure/GDP 21.4% 24.5% 21.6% 20.8% 19.4% 19.3% Health Sector Spending/GDP 1.1% 1.1% 0.7% 1.3% 1.2% 1.2% Health Sector Spending/Total State Budget 5.4% 4.6% 3.1% 6.4% 6.0% 6.3% Health Sector Financing/Approved Health Budget 84% 78% 47% 88% 98% 93% Real growth in health sector spending Memo: Health Sector Spending/GDP projected at appraisal 2.0% 2.5% 3.0% 3.0% 3.0% n.a Note: health sector spending is measured as actual financing, rather than executed expenditures, as financing was significantlybelowexecutedspendingin 1999-2001, resulting inlarge sector arrears. Overall low sectoral financing notwithstanding, Armenia has taken decisive steps towards reforming the health financing system. The purchasing and the provider function are separated, the Ministry has withdrawn from the provision of health care and providers are financially autonomous, receiving public funds only to the extent that they are contracted by SHA. Public funding for health care has been streamlined into a Basic Benefits Package (BBP) financed by SHA, thereby clearly limiting the state's responsibilities with respect to health care provision. While the latter is not entirely based on cost effectiveness considerations, its contents can be considered reasonably rational. Efforts have been made to bring the BBP gradually more in line with available resources, though it still continues to be under-funded. Targeting towards lower income groups has improved in principle with the inclusion of family benefits beneficiaries under the category of vulnerable population for which the benefits package is more generous than for the rest of the population. The project has helped put in place the basic requirements to improve efficiency, transparency and targeting of public spending by helping establish SHA which contracts providers based on the BBP and has become adept at controlling execution of publicly funded health services. A significant amount of work was carried under the project to support the introduction of new and gradually improving provider payment mechanisms. The combination of improvements in the payment mechanisms and somewhat improved budgetary execution during the second half of the project, have led to a system where SHA has managed to clear all past arrears and no new arrears have been accumulated over the past two years. However, the chronic under-funding of the health care system and the continuous large excess capacity, particularly on the hospital side, have not allowed the health finance reforms to have the desired impact on sectoral efficiency, quality and access to health care. Although the reimbursement prices increased during the last years of the project they are still below the cost. Underfunding of the BBP and low reimbursement rates led to rationing and informal payments which then negatively affected the lower income groups' access to care. Armenia was not successful in introducing selective contracting as these attempts met strong political opposition by providers and the Parliament, and the resource allocation across providers is still substantially driven by considerations of equity towards providers, at the expense of efficiency and equity in access to care. The combination of service reimbursement rates which are often deemed to be inadequate to cover costs and the relatively small contract volumes accruing to most providers, have left patients in the need to make formal and informal out of pocket payments to gain access to care to which they are in principle entitled under BBP. It must, however, be recognized that the key factors which have limited the impact of the health finance reforms, such as chronic under-funding, continued excess provider capacity and the need for further - 8 - restructuring of primary and outpatient specialist services, were well beyond the reach of the project. With these limiting background factors in mind, the project can be considered having made a notable contribution towards fundamental changes in the health finance system. The achievements to date undoubtedly harbor the potential to further increase efficiency and equity in access to care, if supported by complimentary reforms on health systems rationalization, organization of primary and outpatient secondary care and higher public funding for health care as is now envisaged under the MTEF and PRSP over the years to come. The overall outcome is rated as moderately satisfactory because substantial structural changes in the health finance system have been undertaken which have improved the management of limited public funding to the sector, but the changes have not had the full expected effects due to the limiting factors described and the fact that the project, by virtue of its design, omitted to focus on improving accountability of contracted providers. Objective 3. Mobilized communities to take an active role in defining local health care priorities and sustaining basic health care services. DO rating: Satisfactory. Target communities were mobilized to support the provision of basic health services by contributing a small amount to the investments in the primary health care centers and have participated in the process of defining health priorities, included in the PHC Development Plans prepared by the retrained physicians. The level of the contribution had to be reduced from the initially planned 10 percent to 2 percent. The second Social Investment Fund project faced similar issues with poorer communities but there the community contribution was picked up by diaspora resources that were not available for the health project. The 76 communities contributed on average US$1,800 each (range US$600 to US$4,000). Compared to other communities, those included in the project were more willing to contribute financially in the future to support the local health center. 4.2 Outputs by components: Component 1: Strengthening Primary Health Care: Highly Satisfactory. The family medicine approach in the delivery of primary care services in Armenia has been successfully introduced and its credibility was established with support from the project. The new and renovated PHC ambulatories, staffed with trained family doctors and equipped adequately have demonstrated the potential for increasing quality, effectiveness and accessibility of care. Training institutions for PHC professionals were established and graduates of their programs cover about 11 percent of the needs for family doctors and 5 percent of PHC nursing staff of the country. These institutions will play a key role in training family doctors to expand coverage nationally, as proposed by the updated PHC development strategy approved in 2003. Sub-Component 1.1. Primary Health Care Providers Training Program: Satisfactory. Armenia was one of the first countries among the Commonwealth of Independent States to establish Chairs of Family Medicine at the State Medical University (SMU), National Institute of Health (NIH) and Basic Medical College (BMC) in 1997. The SMU mainly provides family medicine training for undergraduate medical students and postgraduate residency programs. The NIH mainly provides re-training programs for district therapists, district pediatricians, nurses and midwives as well as residency training. The BMC provides family medicine training for undergraduate nursing students and postgraduate specialization of the family medicine nurses. As a result of the project the three departments have qualified faculty, a well designed training program, and adequate infrastructure. Eight new family medicine trainers and eight family nurses trainers were selected and trained to become trainers in clinical centers, teaching staff of the three institutions have - 9 - attended training activities in Estonia, Netherlands, Norway and Russia, and intensive training courses with participation of foreign consultants were organized in Armenia. The training curricula for family doctors and nurses have been developed. The departments of family medicine received equipment for skill labs and training practices, medical literature, office equipment and furniture. The training facilities at the SMU and hostels for training participants at SMU and NIH were renovated. One difficulty faced by the training program was to provide a good balance of theoretical and practical training. Most of the practical training was carried out in inpatient facilities. This gap was identified during project implementation and a clinical training base was set up at Policlinic 17 in Yerevan. The project supported renovation, provision of equipment and furniture as well as training of family doctors. However, this investment was completed only in the last few months of the project. To strengthen confidence of newly trained PHC providers in everyday practice settings, the training centers have run innovative training-at-site programs. During the project, 116 physicians from target rural ambulatories and Policlinic 17 have completed the retraining program for family doctors. Another 43 physicians have graduated the residency in family medicine. One hundred and twenty-six nurses completed the retraining program for family nurses at NIH and 26 have completed the family nurse residency at BMC. Overall, the number of practitioners was less than planned during project design, as it took more time and effort to establish adequate capacity for training delivery. Although the quantitative targets set initially were not met, the quality of the training programs that are the foundation for continuing training programs, the performance of retrained staff in providing better services and proving the viability of the family medicine approach ­ all this supports the satisfactory rating for this subcomponent. Sub-Component 1.2: Primary Health Care Development Program, Highly Satisfactory. The Primary Health Care Development Program was built on the successful Armenia Social Investment Fund experience, aiming to improve access to essential services in rural areas. The program was closely linked with the health services decentralization policy that transferred the ownership and responsibility for PHC services to local governments and with the PHC training programs, supported by subcomponent 1.1, to ensure that the outfitted rural facilities also have qualified staff. The PHC development program has financed 81 subprojects, compared to 70 envisaged in the initial design. A number of 75 subprojects were implemented in rural communities (covering about 17 percent of population, excluding Yerevan) and six supported the establishment of an urban Family Medicine Center, at the Policlinic Number 3 in Gyumri. A public information campaign and over 150 community meetings supported the process of preparing and selecting subprojects. Facility management boards including representatives from the Marz health department, PHC team, community and the head of the hamaynk (local community) were established to conduct community health needs assessment, submit proposals and business plans and oversee the implementation of the program at the local level. Selection of projects proposed by hamaynks took into account institutional, sustainability potential, health needs, social, participatory, economic and technical criteria. The project supported investment in basic medical equipment for the 81 PHC ambulatories, renovation of 22 facilities, at average cost of US$19.2 thousand, and building of 54 new facilities, at average cost of US$33.3 thousand. Average cost was higher than initially estimated (US$25 thousand). One potential problem related to the upgraded primary care ambulatories was the fact that standard designs for new ambulatories or renovation of existing facilities did not always allow for adjusting construction size to the staff and volume of services required for the size of the catchment population. This poses risks for sustainable maintenance in smaller communities. This lesson was used to in the design of the Health - 10 - System Modernization Project where several standards were developed to take into account the needs of population. Given that the Dutch grant, which became available for the health financing component during the project preparation, allowed to reallocate additional resources to the PHC component, this subcomponent supported the following additional investments: selected rural ambulatories and Marz policlinics were provided with 91 ambulances; the Diagnostic Centers were established in Marz policlinics of all 11 Marzes. The centers were equipped with diagnostic equipment and 105 staff from the centers was retrained. The public health services network was strengthened through provision of modern bacteriological laboratory equipment to 40 local epidemiological centers and 12 portable laboratories to Marz epidemiological centers. By combining several interventions the PHC development program was very successful not only in improving the quality and access to primary care, but also in increasing willingness of communities in decision-making and financing of basic services. Sub-Component 1.3: Primary Health Care Guidelines Development, Highly Satisfactory. For standardization of the work of newly retrained family physicians guidelines for screening and diagnostic procedures, medical case management and practice case management algorithms were developed, based on international good practice and adapted with consultation with Armenian professionals. In total, 13 volumes (127 guidelines) for family physicians and five volumes for family nurses (56 guidelines) were developed. Each book of family guidelines was published in 1200 copies and distributed to health professionals of family medicine departments of Yerevan SMU, NIH, BMC, 30 policlinics of Yerevan and policlinics and ambulatories of 10 Marzes of Armenia. Training seminars were organized for dissemination of newly published guidelines and surveys were conducted to evaluate the use of guidelines by health professionals. The survey results indicated that the guidelines were being used by a very high proportion of practitioners in primary care in their day-to-day work (over 90 percent) and evaluations of medical files have documented improved quality of care in the management of common conditions. Evaluation by users and external peer reviewer have suggested the need to distribute a larger number of copies, update guidelines periodically, incorporate latest evidence based scientific knowledge and also take into account available diagnostic and treatment procedures. Component 2: Strengthening Health Financing System: Moderately Satisfactory. The project has helped introduce fundamental changes in Armenia's health finance system and supported the establishment of SHA as the agency which contracts and pays providers for health services. The component was designed to help develop the technical capacity in SHA and MOH to more rationally plan for sectoral resource allocation, introduce output based provider payment mechanisms and establish a financing agency capable of contracting and paying providers for services. Despite chronic under-funding, erratic resource flows and political interference in resource allocation decisions this was substantially achieved, although performance under various sub-components varied. While there is still ample room to further improve health financing mechanisms to derive at a system which incites providers to increase efficiency and improves equitable access to care, the project helped introduce the foundations of a more rational health finance system on which further reforms can build. One weakness of this component's design was that it focused almost exclusively on the financing side, while ignoring the need to strengthen provider's management capacity, introducing an adequate accountability framework for autonomous providers and to link provider payment reforms with the establishment of quality assurance mechanisms. Although this was apparently a conscious decision by the project team in order to keep the project focused, it clearly limited the impact of the project on provider - 11 - performance. These issues were picked up during the project implementation and addressed under structural adjustment lending program. Achievements under individual sub-components are discussed below: Component 2.1: Establishing the State Health Agency: Satisfactory. The project successfully supported the development of SHA and its ten regional branches into an agency with able technical capacity to budget, contract and pay for health service, which was the main objective of this sub-component. SHA plays a key role in allocating the limited public sector resources for health care across regions, working with MOH on the formulation of the annual health program budgets and subsequently assuring provider contracting, contract performance monitoring and payment processing with providers. The main rationale for the establishment of SHA was to relieve MOH from health care provider payment related administrative functions, while assuring full separation of financing from provision of health care. This has been achieved under the sub-component, which thus justifies the satisfactory output rating. SHA was originally conceived as a semi-autonomous public agency under the Prime Minister's office. As such it was to be governed by a management board comprising key higher level SHA staff assigned by the President but responsible to SHA's director. The main shortcoming of this set-up was that the agency was not subject to any external governance and thus lacking a credible accountability framework. At times this resulted in significant tension between SHA and MOH leadership and calls to bring SHA under MOH's umbrella. Towards the end of 2001, with SAC IV support, an inter-departmental supervisory committee was established to provide an external governance structure and thus improve SHA oversight. In July 2002, SHA was brought under the umbrella of MOH, thereby losing its status as semi-autonomous state agency. At the same time the supervisory committee was also dissolved. While this has been a significant change on legal grounds, it has had little impact on SHA's operations. A provider survey carried out as part of project evaluation indicates that providers have not noticed any change in their dealings with SHA as a result of the agency's being brought under MOH authority. This reflects the fact that SHA was never set up as a strategic purchaser, but as a contracting and payment agency, functions which it continues to exercise as previously. To the extent that MOH is no longer a service provider, the move of SHA under MOH authority has not significantly affected the purchaser provider split which had been the main motivation for establishment of SHA. The experience suggests that the initial establishment of SHA as an independent public agency was somewhat premature, as the country was lacking the necessary regulatory capacity, experience and skills to assure adequate governance over truly autonomous public agencies. Component 2.2: Improvement of Basic Benefits Package Methodology: Moderately Satisfactory. The main purpose of this sub-component was to identify a basic health care benefits package (BBP) consistent with cost effectiveness considerations, the country's epidemiological profile and the limited budgetary funds for the health sector, thereby clearly identifying and limiting the State's obligations with respect to the financing of health care services. A technical working group was charged with the development of a methodology and proposed contents of the BBP on an annual basis, although the main responsibility for this was increasingly assumed by MOH and SHA towards the last two project years, as their technical capacity to do so had improved. The working group reviewed international experience with BBP definition, carried out limited burden of disease and cost effectiveness analysis and then annually developed a program based BBP to guide budgetary allocations. The BBP was submitted to Government and subsequently to Parliament for approval. Since 2001 the Parliament only approves the budget of broad programs but leaves it to Government to formulate detailed program contents or change them. The benefits package developed and proposed by the working group was frequently expanded to satisfy demands from various interest groups. - 12 - During the years of project implementation the type of beneficiaries and the scope of services covered under the BBP changed frequently. For example BBP ensured free health care for children ages 0-7 years in 1997, 1999, 2000, for children 0-15 years in 1998, for children 0-3 years in 2001 and for children 0-6 years in 2002 and 2003; similarly, haemodialysis (as an example) was covered in 1997-1999, and again in 2003. The frequent changes created uncertainties among patients and providers and left room for abuse. Overall there does not seem to have been a clear trend towards improving the scope of services covered under BBP towards increased cost-effectiveness over the duration of the project. Rather, services were dropped when budgetary resources became scarcer (2000, 2001) and added again when funding availability increased. On the other hand, a significant step to improve BBP targeting was taken in 2001, when Government decided to expand BBP eligibility to individuals who receive means tested Family Poverty Benefits (FPB). Also, the Government did reduce the hospital treatment programs for certain infectious diseases and mental health that could be effectively managed on outpatient basis and clarified the definition of emergency care that helped to cut over-charging and reduce expenditures. While data to measure the effectiveness of this step remains still scarce, a comparison of household survey data (LSMS data) from 1998 and 2001 indicates that the health services utilization rate among those with family poverty benefits was higher (32 percent) than among those not eligible (29 percent) in 2001, with a marked increase in utilization among the lower income groups in rural areas between 1998 and 2001. Data also show a slight decrease in informal payments during the same time period. This would thus suggest that improved targeting of BBP benefits had a positive initial impact on lower income groups' access to health care. To what extent this impact was further reinforced during 2002 and 2003 as further experience with reaching FPB beneficiaries was gained needs to await availability of new household survey data. While the current BBP can be considered reasonably rational (free PHC and obstetric-gynecological services for all, free outpatient specialist and diagnostic services and hospitalization for vulnerable groups in including FPB beneficiaries, free emergency care for all), the extent to which its contents for secondary and hospital care is driven by cost effectiveness considerations and transparency remains somewhat questionable. It must, however, also be recognized that Armenia is among only a few countries in the ECA region which have taken the decisive and difficult step to define a positive (rather than a negative) list of services covered under public funding and that there is essentially no country which has a BBP that is based entirely on cost-effectiveness criteria. Despite increases in public sector spending for health care over the past two years, the BBP remains under-funded, leading to provider reimbursements which are some times below costs and concomitant informal payments with ensuing access barriers for low income groups. To help address some of these issues SHA plans to use further budgetary increases to increase payment rates for existing services, rather than expand the scope of BBP over the next few years. Thus, while the working groups and technical assistance provided under this component did make a significant contribution towards shaping the dialogue on state funded health care benefits based on technically sound arguments, the final outcome of the BBP has remained subject to political bargaining. The sub-component is therefore rated as only moderately satisfactory. Component 2.3: Improvement of Provider Payment Methodology: Moderately Satisfactory. This component helped develop provider payment mechanisms aimed at assuring that obligations towards providers fit within the available resource envelope and that providers are paid on an output rather than input basis. This has resulted in a provider payment systems where hospitals are paid on a per case basis, within an overall global budget per program, rural ambulatories and policlinics are paid on a capitation basis (adjusted for broad demographic characteristics) for primary and dispensary care and on a fee for service basis (within program ceilings) for consultations with narrow specialists and diagnostic services. This system has allowed to keep public sector commitments towards providers within the available budgetary enveloped, but it did not stop SHA from accruing arrears through 2001, because budgetary execution remained poor through that year. Over the last two years, however, budgetary execution has - 13 - markedly improved. Combined with a cautious approach to contracting by SHA (SHA contracts for less than the approved budget with an option to adjust the contract during the course of the year if additional funds become available), this has allowed SHA to fulfill its contractual obligations towards providers without incurring any arrears, while previously accrued arrears were cleared over the course of 2002. Achievements to date notwithstanding, the current provider payment mechanisms will require further strengthening to achieve greater provider responsiveness in terms of efficiency and quality of care. This will require linking payments to performance targets and supporting improved payment mechanisms with service delivery reforms (e.g. clear separation of primary care providers from narrow specialists within policlinics in urban areas, primary care physicians as gate keepers, integration of outpatient specialist care with hospital care; rationalization of service provider network) which will allow payment mechanisms to have the necessary effects on efficiency and quality of service provision. Planned reforms of the delivery network which will separate family physicians from specialist outpatient care, while integrating the latter with hospital care, further strengthening and expansion of family medicine providers and introduction of a gate keeping system should help remove lingering perverse incentives in the payment system to policlinics. Further efforts will need to be made to expand official co-payments for SHA financed benefits, while taking more aggressive action to reduce informal payments. The technical working group, supported by Dutch grant funded technical assistance, which supported SHA with the development of provider payment mechanisms and provider contracts during the first few project years, also worked on developing proposals to introduce co-payments for state guaranteed services and selective contracting. In 2001 as part of a commitment under SAC IV, the SHA tested selective contracting and co-payments for maternity services. However, strong opposition voiced from providers and Parliament led the Government to abandon the ideas in 2002. Co-payments for state funded hospital services were again introduced in Yerevan hospitals on a pilot basis in the fall of 2003, with the objective of evaluating the experience in view of subsequent broader introduction of co-payments. The sub-component's overall performance is rated as moderately satisfactory. While it has helped build capacity in SHA to introduce and manage new provider payment mechanisms and contracts which have allowed for an overall more rational public sector funding system, further improvements and supportive organizational reforms are required to allow new payment mechanisms to have the desired effects on provider behavior. Component 2.4: Financial Information System: Satisfactory. This sub-component was originally designed to improve the efficiency of health financing management through the development of new medical and financial reporting forms, automation of data collection and analysis, and automation of agency payments to providers. Since SHA was established to be operational after less than one year of project effectiveness, it was decided to develop an interim system as quickly as possible, while allowing for more time to then develop a fully fledged financial management information system. The interim system was developed and introduced during the first project year, while tender documents were developed for the more developed system. When the interim system started experiencing problems, it was further upgraded with the assistance from USAID in 2001-2002. Today the system is well developed and supports effective financial management by SHA, which was the original objective of this component. It allows to monitor provider service provision and contract based expenditures, calculate resource allocations across providers and carry out statistical analysis of financial and activity information. It is better developed on the hospital than on the PHC side, though SHA already has clear plans to further upgrade it to better monitor and analyze PHC performance. - 14 - During the course of project implementation, the Government took a decision that it wanted to pursue an integrated health information system (HIS) which would combine health finance, health resource and epidemiological information and the scope of the sub-component was thus expanded to support the development of the HIS. The operational plan for the system was developed with assistance from USAID which had initially also agreed to finance the development of the system, while IDA resources would have financed hard- and software. When USAID withdrew its support, the Bank agreed to finance also the system development. The development of the system experienced delays because no responsive bids were received during the first tender. During the last project year, however, the system was successfully developed and hardware was purchased to test the system at MOH, SHA and in three Marzes, including two hospitals and two policlinics in each Marz. At project completion the system was still being tested in the three Marzes. The Government planned on a full evaluation of the system within a year of project completion and, subject to positive piloting results, expansion of the system to additional Marzes and providers. IDA's recently approved Health System Modernization project will build on these achievements. Component 3: Project Management: Satisfactory. Overall coordination and administration of the project was assured by a PCU which was staffed by project management as well as technical staff to support implementation planning of key project activities, particularly on the primary health care development side. The legal status of the PCU changed several times during the course of the project, until it was finally brought under the umbrella of MOH as a state agency. The PCU reported to a project supervisory board chaired by the Minister of Health and with representatives of the Ministry of Finance and other core agencies. The PCU benefited from continuity in the Director position, proactive management and an overall well qualified and committed group of management and technical staff. A substantial amount of staff turnover about half way through project implementation resulted in some disruption, although transitions were well managed by the Director. The PCU interacted well with the technical working groups and assured that the latter's work was well aligned with successive Ministerial administration's priorities, while remaining focused on project objectives. The PCU played a key role in the planning and organization of PHC training and provided an important bridge between technical working groups, training institutions and concerned Ministerial departments. The PCU was also directly engaged in PHC related community mobilization activities and regularly interacted with local governments to assure their support for project activities. The PCU at times experienced problems in procurement and contract management, particularly with design, construction and supervision of civil works. This was due to a combination of an immature independent construction industry, particularly in the Marzes, PCU staff's limited experience with procurement and contract management, unclear assignment of responsibilities on contract management across PCU staff and poor filing. A supplementary independent procurement review carried out in June 2003 found that the shortcomings had been addressed. 4.3 Net Present Value/Economic rate of return: Quantitative cost-benefit or cost-effectiveness analysis was not undertaken during project preparation or appraisal. The project's focus on primary health care was based on the rationale that improved quality and access to primary care would benefit a larger share of the population than a focus on secondary and hospital care; the potential of strengthened PHC to improve access to care by virtue of its proximity to the population and, the pressing need to improve the balance between primary and secondary care in Armenia, in view of improved cost-effectiveness of primary care. These considerations all remain valid to date. In particular, the beneficiary assessment has shown that the population's access to care in target villages improved, out-of-pocket payments were reduced, referral and self-referral to specialist care decreased and patient satisfaction increased compared to other villages. Furthermore, cost-benefit analysis of the proposed expansion of the primary health care component under the recently approved Health System Modernization project, indicated that these interventions have rate of return well beyond the opportunity - 15 - cost of capital. The direct benefits of the health finance component are more difficult to quantify, although international evidence suggests that the type of systemic reforms supported under the project harbor the potential to substantially increase sectoral efficiency and contain costs. 4.4 Financial rate of return: N/A. 4.5 Institutional development impact: The project has had a substantial impact on institutional development in Armenia's health sector. The project has helped establish some of the first family medicine chairs in the countries of the former Soviet Union. The Chairs have played a key role in initiating the transformation of the Armenian PHC system into a more integrated family medicine based system. The Chairs have developed credible capacity to provide both residency and in-service retraining in family medicine and will continue to play a crucial role as the family medicine system will be rolled out throughout the country in the coming five years. Outcome indicators pertaining to patient confidence in primary care providers, physician and patient self- referral rates show a marked improvement among family physicians and the communities they serve, compared to control groups and baseline surveys, pointing to substantially increased capacity of physicians trained under the project. This clearly is a demonstration of the increased capacity at the provider level. The SHA, established with direct support of the project, has developed substantial capacity in provider contracting and provider payments. The project has also helped strengthen the capacity of SHA and MOH in program based budgeting and resource allocation planning. While not all of the work undertaken by the BBP and provider payment mechanisms working groups may have resulted in direct improvements in the health finance system, the project has nevertheless allowed for the development of substantial analytical capacity in SHA, MOH and among working group members who continue to work in the health sector in various capacities. 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: The Russia financial crisis of 1998 negatively affected GDP growth and the country's fiscal performance. This in turn led to continuous low public spending on health care and highly irregular and unpredictable resource flows to the sector and ultimately providers until at least mid-2001 (see table 1 in section 4.2). It also resulted in counterpart fund problems during the early project years. The slow down in economic growth and decline in remittances may also have had an impact on communities' inability to make expected financial contributions towards health facilities rehabilitations during the first three project years. Another factor which negatively affected project implementation was the conservative mentality in the medical community with the ensuing challenges to gain acceptance of the concept of family medicine among providers (particularly specialists), and to a certain extent also among the population, particularly in urban areas. A third factor which rendered implementation difficult at times was the poor performance of civil works contractors that can be ascribed to a still maturing private sector construction industry, particularly outside Yerevan. 5.2 Factors generally subject to government control: The project benefited from strong support and active interest of the Government throughout its implementation. The PCU was required to regularly report to the Cabinet of Ministers on performance and annual work plans were subject to Cabinet approval. The twice changing legal status of the PCU at times - 16 - brought some disruption and uncertainty for its staff. The establishment of a project management board with representation from five key ministries once the PCU was moved from state enterprise to joint stock company, and subsequently state agency status substantially improved overall project oversight and guidance. Two factors which proved taxing for the PCU were the onerous reporting requirements and frequent changes in rules and procedures applicable to PCUs, and the multiple audits by a number of different auditing bodies (Control Chamber of Parliament, Ministry of Finance, Government audit department, Presidential audit department, and intra-ministerial audits). The audits were labor intensive, often overlapping and more inclined to check for procedural errors than project effectiveness or appropriate funding use. The audits continuously required the PCU to spend a considerable amount of time and resources on them rather than on more substantive project implementation issues. The project also enjoyed close support from regional (marz) and municipal governments, which substantially contributed to its success at the local level. The primary health care component benefited from strong ownership and support of the Directors of the training institutions which was key to its success. The health finance component's performance was negatively affected by the chronic underfunding to providers, a fact which the Government could have controlled more effectively by: (i) allocating a higher share of public spending to health care; (ii) further limiting the scope of the BBP; (iii) taking more decisive action to rationalize the provider network, particularly in Yerevan, so as to allow for improved financial viability of remaining providers; and (iv) move more decisively on selective contracting and co-payments, though it must be recognized that political realities were such that more decisive action in many of these areas would indeed have been extremely difficult. 5.3 Factors generally subject to implementing agency control: Despite relatively frequent changes in the Ministry of Health (MOH) leadership during the life of the project, the project generally enjoyed strong support and a good working relationship with the Ministry. The frequent change in the Ministerial leadership (the project was prepared and implemented under five different Ministers) at times resulted in implementation slow downs during transition periods, and limited changes in policy direction of which the most important were the above described changing opinions about the status of SHA, and changes in the approach to selective contracting and co-payments. The eventual move of the PCU under the governance of the Ministry of Health helped to clarify the relationship between the Ministry and the PCU, and substantially contributed to improved collaboration and Ministerial support. The project benefited from strong and committed leadership at the PCU throughout preparation and implementation. There was only one change in PCU director throughout project preparation and implementation. Continuity in strong PCU leadership proved particularly important in the face of frequent changes in the ministerial leadership, and was an essential part of successful project implementation. The PCU also benefited from a strong team of technical staff particularly in the primary health care development and monitoring and evaluation units. Their regular and closer interaction with training institutions, concerned ministerial departments, local governments and communities were essential to successful implementation of the PHC program. Overall, the PCU often also acted as a technical resource to support MOH in the planning and implementation of reforms, particularly on the PHC front. The high quality of the monitoring and evaluation system developed during the second half of the project, and the support and interest by MOH in the results of the final evaluation further attest to the technical capacity developed by the PCU, as well as the strong ownership of the programs supported under the project by MOH and other concerned implementing agencies. - 17 - 5.4 Costs and financing: The original project cost estimate was US$12.1 million, of which US$10.0 million equivalent was to be financed by the IDA credit. When Dutch grant financing became available for technical assistance and some training in support of the health finance component, a decision was made to reallocate IDA credit funds, which had been budgeted for this purpose towards the first component. Together with substantially lower costs for computer hardware and software under the second component, this resulted in a reallocation of US$1.7 million of IDA funds from the second towards the first component. Overall, total project costs for the second component amounted to less than 40 percent of the appraisal estimate (excluding Dutch grant funding). The additional allocations towards the first component allowed to cover higher than anticipated costs for rehabilitation of primary care facilities, and to reduce the counterpart and community contributions for civil works, a step which helped substantially accelerate implementation of relevant activities under the first component. In addition, a decision was made to finance the equipment of regional diagnostic laboratories and purchase ambulances rather than four-wheel drive vehicles for the rural ambulatories, which were supported under the project. Overall, final costs under the first component were about 8 percent over appraisal estimates, with an IDA contribution of over 20 percent above appraisal estimates, as a result of reduced Government contributions, higher construction unit costs, and financing of supplementary activities. In the face of serious counterpart funds constraints after the Russia crisis, the counterpart funds contribution towards civil works was reduced from 10 to 8 percent. Similarly, community participation in health facilities rehabilitation was reduced from 10 to 2 percent, as communities were not able to come up with the initially agreed upon participation in the aftermath of the crisis. Ninety-nine percent of credit proceeds were disbursed and a balance of SDR89,458.85 was cancelled at the end of May 2004. 6. Sustainability 6.1 Rationale for sustainability rating: The sustainability of project benefits is rated as likely. The Government is very committed to build on achievements of the completed project and continue to improve health sector financing as well as modernize provision of primary care services using the family medicine approach. This commitment is demonstrated by the strategy updates approved by the Government and by the follow-up Health System Modernization Project. As part of the Poverty Reduction Strategy approved in 2003, the Government set for the health sector objectives of increasing accessibility and quality of health services, especially for the poor. Enhancing accessibility is expected to be achieved through ensuring sustainable public financing at a level affordable for the economy; relevant redistribution of intrasectoral allocations to PHC as the more cost-effective care modality, and optimization and efficiency improvement of health care provision. Under the current sustained GDP growth scenario, the Government has committed to increase public funding from the current 1.3 percent of GDP to about 1.9 percent of GDP in 2006 and 2.1 percent in 2009. The PRSP contains two important aspects with regard to sustainability of the project: (i) increase of the share of Government health sector spending for PHC is projected to increase to 40 percent in 2006, continuing the upward trend experienced during the project (from 11.3 to 21.9 percent between 1997 and 2001) ensuring the sustainability of the PHC provision by newly trained providers and improved infrastructure; (ii) improved funding levels will also be used to better align health care reimbursement rates with the actual costs rather than to expand the basic benefits package of services ­ this will ensure that the SHA contracts will get closer to sustainable financing of health services for the poor and cover the costs of providers. Also, the Government medium term expenditure framework (MTEF) is consistent with the objectives outlined in the - 18 - PRSP projecting that health sector funding is the fastest growing single component (besides education) in the planned multi-year budgets. This strongly reinforces the Government's commitment to sustained improvement in the health sector. The training of family medicine providers was designed from the outset to take into account downstream sustainability issues. The retraining effort of existing health professionals that is a major one-time investment was supported with the credit funds and will continue to be supported in the follow-up projects. The residency training, i.e. training to address the natural attrition in the profession, has been financed by the Government. This will continue to be a Government responsibility once the retraining program is completed. In terms of intellectual capital invested into the faculty of the training institutions as well as into newly trained family doctors, there are several developments that indicate that this has been a sustainable investment: (i) Family Medicine is recognized by the Government as distinctive specialty and has the same professional standing as other specialties; (ii) Armenia has joined the World Organization of the Family Doctors and is a recognized member of the international peer community; (iii) newly trained family doctors in Armenia have established a Family Doctors' Association that represents the interests of this new profession and also engages in quality assurance and continuous training issues. Some of the issues that became evident during the implementation of the project that have impact on sustainability were addressed in parallel through the policy dialogue under the Structural Adjustment Credit program and in the preparation of the follow-up Health System Modernization Project. These steps included the following: (i) Government has taken proactive steps to consolidate and optimize excessive health sector infrastructure, in particular hospital and policlinic buildings; (ii) new regulations have been adopted for improved financial management and accounting in public hospitals and relevant training was provided to accountants; (iii) the Health System Modernization Project will strengthen Government's institutional capacity for health sector oversight, accountability and quality assurance; and (iv) the PHC strategy update recognizes the need to define legal status of family doctors, make them managerially autonomous from specialist care and introduce open enrollment - these policy steps provide an important policy environment for continued implementation of PHC reform and the proposed Health System Modernization program supported by the Bank. These measures are all intended to improve the performance of the health sector and ensure sustainable operation. 6.2 Transition arrangement to regular operations: Operating cost of the upgraded primary care services are covered by the contracts with the State Health Agency and contributions from the communities. The salary component of the capitation payment received by PHC providers is higher if the physician has been retrained as a family physician, and also according to the remoteness of the PHC facility. Retrained physicians can also receive additional payments for treating a broader range of cases. However, given the overall current low level of payments, these are relatively small incentives for retraining or working in rural areas. As indicated above, the Government is already financing the residency training of family doctors and family medicine nurses through state order that will be the main path of training PHC providers after the transition period. 7. Bank and Borrower Performance Bank 7.1 Lending: The Bank's performance during project preparation was satisfactory. The project was in line with stated - 19 - Government reform priorities and strategies in the health sector, focusing on primary health care and health finance. Preparation was supported by analytical work that was used to guide project design. The project was consistent with the objectives of the 1997 Country Assistance Strategy and built on the good experience of the Armenia SIF project. Changes to the project design made during implementation were in line with the original development objectives and the main policies. However, the original assessment overestimated the potential impact of project interventions on resource allocation decisions for the sector, as well as the risks of setting up an autonomous health financing institution, without an adequate framework of governance and accountability. These issues were duly taken up during the project implementation and addressed in parallel under the Structural Adjustment Credit program. 7.2 Supervision: Performance of the Bank during supervision was satisfactory. Technically sound and frequent supervision conducted by the project team, with high quality contribution provided on an ongoing basis by the local project officer, allowed for timely identification of the problems and development of viable solutions. The Bank demonstrated flexibility in the face of fiscal crisis experienced during the project, allowing to increase the IDA financing share for civil works and to lower community participation from 10 percent to 2 percent of the cost. Substantial support was provided to the Borrower to establish and maintain a very good monitoring and evaluation system as well as to keep major stakeholders and donors informed. The Bank was involved in extensive donor coordination activities. One tangible result was the contribution of the USAID supported Armenia Social Transition Project to the development of unified family medicine curriculum, endorsed by the Ministry of Health; this assistance complemented the support provided by the project to the establishment of training capacity for family medicine. The Bank made good use of adjustment loans to support policy reforms which the investment project was trying to promote. Good cooperation of the Health and structural adjustment lending teams ensured that the health reform agenda and budgetary issues were adequately addressed in the adjustment lending conditionalities. Examples include elimination of arrears in the health sector, adoption of rationalization plans, establishment of an inter-department supervisory committee for SHA; capping of hospital contracts; introduction of co-payments and of adequate accounting requirements for hospitals. 7.3 Overall Bank performance: The Bank's overall performance is rated satisfactory. Borrower 7.4 Preparation: The borrower's performance during project preparation was satisfactory. The borrower showed strong leadership and willingness to proceed with sectoral reforms during project preparation, taking decisive steps to initiate reforms prior to project start-up. The borrower actively engaged in project preparation and policy dialogue throughout project preparation. Two technical working groups were established by the MOH to refine policy reforms measures and define implementation plans which were to be supported through the project. The working groups were supported by international technical assistance financed under a PHRD grant and supported by a Project Coordination Unit initially established outside MOH. The MOH organized consultative meetings with stake holders at central and local levels to discuss the proposed reform concepts to be supported under the project. - 20 - 7.5 Government implementation performance: The key issue which afflicted the health sector during the entire project implementation period was the severe shortage of public sector funds for health care. This was due to a combination of need for fiscal restraint and low priority accorded to social sector spending. During the first three project years, health sector budget execution consistently fell substantially below approved budgets and the flow of funds was highly erratic, which made it impossible for SHA to properly manage provider contracts. The situation was at times further aggravated by Government's decisions to augment the BBP proposed by the technical working groups as a result of interest group pressures. On the other hand, the Government, through the establishment of the project supervisory board, showed a continued strong interest in the project's activities, performance and achievements. The board provided overall much appreciated guidance and direction to the PCU and paid close attention to implementation progress and achievements. Overall the Government's implementation performance is rated as satisfactory. 7.6 Implementing Agency: The performance of the Ministry of Health, the State Health Agency and the training institutions involved in family medicine training were all satisfactory. The Ministry of Health, including successive Ministers, took a strong interest in the project and closely followed the work carried out by technical working groups, as well as the achievements on the family medicine front. The State Health Agency developed into a body quite adept on provider contracting and increasingly internalized the work initially carried out by the provider payment and BBP working groups. The training institutions showed strong commitment and leadership during the establishment of the family medicine chairs, and the subsequent development and implementation of the family medicine training programs. As described above, the project benefited from overall strong performance of the PCU which was adequately staffed and resourced throughout the project. 7.7 Overall Borrower performance: The borrower's overall performance during project preparation and implementation was satisfactory. The borrower showed substantial commitment to implement the project support reforms, particularly on the primary health care front. The project overall enjoyed strong support from the implementing agency leadership despite frequent changes in the latter. 8. Lessons Learned The following lessons can be drawn from the implementation experience of the Bank's first health project in Armenia: Ø Introduction of provider contracting and changes in provider payment mechanisms need to be supported by substantial efforts to increase information flow, provider accountability, quality assurance and appropriate service delivery reforms to achieve the expected results. Ø Unless public financing constitutes a significant share of provider income and the purchaser has the ability to enforce accountability, new provider payment systems are unlikely to have a significant impact on provider performance and efficiency. Ø In a system with substantial excess capacity, payment reforms and provider autonomy alone will not result in significant efficiency improvements nor systems rationalization, if not supported by more decisive measures to concentrate resources on selected providers or active supply side interventions to - 21 - reduce over capacity. This lesson has been taken into account in the design of the Armenia Health System Modernization Project. Ø Need to supplement class room and hospital based family medicine training with clinical training to allow family doctors to gain hands on experience in clinical setting before they are on their own ­ lesson integrated into updated design of Government family medicine expansion program for 2004-2009 which will be supported by the Bank's Health System Modernization Project. Ø Need to improve the regulatory framework for family medicine in order to ensure that newly trained family doctors and family medicine nurses can practice their new skills and family medicine concept. Legal status of family doctors needs to be defined. For family doctors to be able to attract sufficient number of patients to their practices, the rigid population assignments based on residence need to be relaxed. Family doctors should also be able to contract the SHA directly and the incentives need to be improved for physicians to be retrained as family doctors. These lessons have been taken into account and they will be addressed by the effectiveness of the follow-up Health System Modernization Project. Ø The project demonstrated that any design weaknesses and/or emerging policy issues can be effectively addressed during implementation by using other Bank instruments, such as a structural adjustment lending program. This requires proactive supervision and continuous engagement in policy issues by the project team. 9. Partner Comments (a) Borrower/implementing agency: The following comments of the Government have been received from H.E. Norayr Davidyan, Minister of Health, in a letter of June 10, 2004. The original letter is available in the project files. The World Bank supported Armenian Health Finance and Primary Health Care (PHC) Development Project initiated in 1997, was completed on 31st December 2003. The project significantly contributes to the success of the country current health reforms implementation and developments. Health Financing and PHC development sectors were selected as targets for the Project, which in the past and at present are considered to be as the most important trends of the health reform. With support of the project, the MOH undertook initial steps in laying of a solid foundation for introduction and gradual development of family medicine in Armenia. Appropriate preconditions have been created to render high quality primary care to the population, especially to the vulnerable groups. Eighty-one offices for family physicians have been reconstructed, equipped and supplied; personnel have been retrained. Acquiring new skills, physicians now work in a more comprehensive and satisfactory manner. Communities where the micro projects have been implemented have seen improvement of PHC accessibility, decrease in referral rate to narrow specialists, enhancement of population satisfaction with PHC services offered by the trained physicians. Although at the beginning of the program, it was decided to improve building conditions of selected ambulatories and provide them with furniture and medical equipment, later, in order to make the program's impact more comprehensive and systematic some changes have been introduced to the process of its implementation. Particularly, the selected ambulatories and Marz polyclinics were provided with ambulances equipped with necessary and up-to-date devices. In all Marzes, 11 Diagnostic Centers were established as a part of Marz polyclinics' structure and were equipped with diagnostic equipment. The medical staff of the Centers was retrained. Forty Hygiene and Epidemiological Centers of all Marzes were provided with modern bacteriological laboratories and Marz Hygiene and Epidemiological Centers with 12 portable laboratories. That was considerable and tangible - 22 - input in reforming and modernizing the rural primary health care system of Armenia. These activities have also played an important role in using community resources, and the population now realizes the need for personal participation in solving issues related to their own communities. The significant investments have been made in family medicine (FM) training infrastructure organization. Chairs of Family Medicine were established at the National Institute of Health, Yerevan State Medical University and Basic Medical College; family medicine practical training center was established-Yerevan Polyclinic #17. Statements of the Family Physician/Family Nurse were developed and approved, based on which family medicine training curricula have been worked out. The FM training activities benefited greatly by the specialized literature, clinical guidelines, waxwork, audio and video materials, and computers and other hardware provided to the chairs by the Health PIU. Developed within the framework of the Project FM clinical guidelines, by this time, are the only clinical standards applied in the practice of Armenia and that is why their role is very noteworthy. All above is invaluable input for further PHC development projects. The Project played a key role in differentiation of health care providers and financing authorities. With creation of State Health Agency (SHA) as purchaser of publicly financed health care, it was become possible to pay more attention to the control of services delivery and efficient use of public funds. The financing was shifted from line item budget to contract based payments for a defined package of basic health care benefits (BBP). New reporting forms for health providers have been developed; automated systems for information collection and analysis on the basis on these reports have been introduced. That played an important role in the process of health care services planning and cost effectiveness studies. In addition, the Project assisted the MOH in providing rationale for annual health care target programs setting and estimating health budget, in improving provider payment mechanisms and other issues of health care financing and regulatory/legislative reforms in that area. The major effort towards improvement of health care provider payment system has been the development and introduction of global budget approach, allowing containment of department accumulation and providing incentives for the management of health care facilities to improve efficiency of financial management. With support of the Bank's first health project, the MOH undertook first steps to establish a health management information system that integrates information from three vertical management information systems - health financing system, public health surveillance system and health information system. IT hardware and software had been purchased for health facilities, Marze health departments, State Health Agency and marze SHA branches, State SanEpid Inspection (SSEI) and marze SSEI branches, Health Information and analytical Center and MOH. Health information system software packages have been developed at all levels of health care delivery and for regional (marz) and national levels for proper health system management and policy development. According to Health Financing and Primary Health Care Development Program evaluation results, all stakeholders expressed opinion about usefulness of the Project and adequateness to the country needs. They noted that the project had set the right goals, reflecting the most crucial challenges faced by the health system of Armenia both at project inception and today. Communities considered that the Project had contributed to the life of their hamaynk and had greatly supported an improvement in the quality and affordability of health care. The heads of a number of target hamaynks mentioned that the project is beneficial not only to their communities, but also residents of neighboring ones. Main constraints encountered included that the introduction of family physicians does not enjoy adequate support from the health financing and legal/regulatory systems. Trained family physicians, which should be rendering a broader circle of services compared to traditional PHC providers, did not have the necessary incentives, because BBP financing is extremely insufficient, and the capacity of family physicians to render paid services was limited. Stakeholders mentioned, as obstacles, the insufficient duration of the training - 23 - program, the lack or extreme scarcity of access to practical skills, the absence of a clinical education bases in the past, as well as the scarcity of conditions (the number of classrooms and patients) necessary for the training process. The experience gained in the process implementation of the Project is serving as a significant factor for successful implementation of further objectives set by the Ministry of Health and provided valuable input in the preparation of the Armenian Health Project 2. (b) Cofinanciers: N/A. (c) Other partners (NGOs/private sector): N/A. 10. Additional Information N/A - 24 - Annex 1. Key Performance Indicators/Log Frame Matrix Outcome / Impact Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate Adequate vaccination coverage (target: at 95% 2003 data are: Polio 96%, DTP 93.3%; least 95%). Measles 93.8%, TB 92% Decrease in incidence of vaccine preventable same as appraisal diseases evaluated based on internationally 2003 incidence rates per 100,000 children criteria (WHO-HFA 2000). are : Polio 0; Diphteria 0, Tetanus 0; Pertussis 0.4; Measles 0.1, TB 5.8 compared to baseline Polio 0; Diphteria 0, Measles 63, Pertussis 0.5, Tetanus 0.1 Declining referral rates from newly trained same as appraisal PHC teams (target to achieve 15% referral Traget Practices: 12% of new and 4% of old rate; starting 1/99). patients referred at time of impact assessment, compared to 29% and 21% in non target practices Improve access to PHC providers in target same as appraisal 20% self referrals in target sites vs 38% at areas; decrease of self-referrals to specialist non target sites; first contact PHC provider care; decrease in no-diagnosis cases for 76% of patients in target sites; 58% in non target sites;. Beneficiary Assessment Data show greater same as appraisal 98% of population trust their PHC provider in accessibility to services, perceived as having target areas vs 94% in non-target areas; 93% better quality, in intervention areas. of respondents received advice on health behaviour from PHC provider vs 83% in non target areas, significant improvements in other areas Improved management of PHC facilities in same as appraisal 55 revised development plans reflect target areas; percentage of practices meeting physicians better understanding of practice development indicators to be set forth in the environment performance contracts (12/99; 06/01). Adequate public sector health spending as % 2002: 1.3%; 2003: 1.3% 2001: 1.34% of GDP, 2002: 1.2% of GDP, of GDP (2% in 1998; 2.5% in 1999; 3% in 2003: 1.3% of GDP 2000; 3% in 2001). SHA assuming full functions (01/99). same as appraisal SHA is carrying out all functions Providers are paid on a timely basis (less same as appraisal No arrears accumulation, but timely payment than 30 days following receipt of correct is still an issue payment information). Increased efficiency and transparency of same as appraisal BBP is overall rational, though is not based BBP, based on the Bank review and entirely on cost effectiveness considerations. comparison with the BBP of a year before One problem is that the contents frequently (12/97, 12/98, 12/99, 12/00). changes, causing confusion among providers and beneficiaries. Taregeting has improved with introduction of recipients of family benefits among vulnerable group Increased efficiency of health services by targets not updated ALOS 11 days in 2003; bed occupancy rate shortening of ALOS (15 days in 1998; 14 37% in 2003 days in 1999; 13 days in 2000) and increasing bed occupancy rates (target 65% in 2000). There will be no payment arrears to same as appraisal there were no payment arrears in 2003 and providers in accordance with the provisions all old arrears were cleared in 2002 of their contract (target 0%). - 25 - Output Indicators: 1 Indicator/Matrix Projected in last PSR Actual/Latest Estimate Number of PHC Providers Trained targets not defined Family Physicians Retrained (NIH/SMU): 116 of which 8 are trainers Family Physician Residents Graduated (NIH/SMU): 43 Family Nurses Retrained (NIH): 130 of which 8 are trainers Family Nurse Residents Graduated (BMC): 26 Number of PHC Centers Rehabilitated and 75 rural 2 urban Rural ambulatories: 80 Equipped. Urban PHC Centers: 2 Number of Services Complementary to PHC diagnostic centers 11, mobile sanepid Diagnostic Centers Established: 11 established: centers 12, diagnostic center staff trained Diagnostic Center Staff trained: 105 105, regional sanepit stations equipped 40 Mobile Sanepid Centers equipped: 12 Regional Sanepid Stations equipped: 40 Number of PHC Guidelines Produced targets not defined Volumes of Family Physician Guidelines: 13 Actual Number of Guidelines Produced: 127 Volumes of Family Nursing Guidelines: 5 Actual Number of Family Nursing Guidelines Produced: 56 Number of SHA branches functional after 10 SHA Center: Completed being rehabilitated and equipped. SHA Regional Branches Functional: 10 SHA Regional Branches Equipped: 10 SHA Regional Branches Rehabilitated: 9 Percentage of community contributions to 100% of community contributions collected 100% of community contribution was PHCDP-financed projects (target 10% of (2% of total costs) collected (totally 75 communities made the total). contribution Us$144,833. Average contribution is US$1800, maximum contribution is US$4000 and minimum contribution is US$600). Number of PHC Development Plans Signed. no target Currently 55 Development Plans have been revised and sigend. Evaluation of the project included 27 plans. Overall, the objectives stated in the plans were met. More info in the project evaluation report. Number of PHC Providers Currently no target Retrained Family Physicians (NIH/SMU): Practicing in PHC Settings (i.e. would not 114 of which 8 are trainers include unemployed Residents, those Family Physician Residents (NIH/SMU): 15 physicians which are working as specialists Family Nurses Retraining (NIH): 126 of or have left the profession). which 8 are trainers Family Nurses Residents (BMC): 10 1End of project - 26 - Annex 2. Project Costs and Financing Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.00 2.60 0.30 0.00 2.90 (0.00) (2.10) (0.20) (0.00) (2.30) 2. Goods 0.80 0.40 1.60 0.20 3.00 (0.80) (0.30) (1.60) (0.00) (2.70) 3. Services 0.00 0.00 3.50 0.00 3.50 (0.00) (0.00) (3.50) (0.00) (3.50) 4. Miscellaneous 0.00 0.00 2.70 0.00 2.70 (0.00) (0.00) (1.50) (0.00) (1.50) 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 0.80 3.00 8.10 0.20 12.10 (0.80) (2.40) (6.80) (0.00) (10.00) Project Costs by Procurement Arrangements (Actual/Latest Estimate) (US$ million equivalent) 1 Procurement Method Expenditure Category ICB NCB 2 N.B.F. Total Cost Other 1. Works 0.31 2.82 0.10 0.00 3.23 (0.28) (2.45) (0.09) (0.00) (2.82) 2. Goods 2.88 0.75 0.90 0.00 4.53 (2.85) (0.73) (0.86) (0.00) (4.44) 3. Services 0.00 0.00 1.34 0.00 1.34 (0.00) (0.00) (1.34) (0.00) (1.34) 4. Miscellaneous 0.00 0.00 0.75 0.00 0.75 (0.00) (0.00) (0.75) (0.00) (0.75) 5. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) 6. Miscellaneous 0.00 0.00 0.00 0.00 0.00 (0.00) (0.00) (0.00) (0.00) (0.00) Total 3.19 3.57 3.09 0.00 9.85 (3.13) (3.18) (3.04) (0.00) (9.35) 1/Figures in parenthesis are the amounts to be financed by the Bank Loan. All costs include contingencies. 2/Includes civil works and goods to be procured through national shopping, consulting services, services of contracted staff of the project management office, training, technical assistance services, and incremental operating costs related to (i) managing the project, and (ii) re-lending project funds to local government units. - 27 - Project Financing by Component (in US$ million equivalent) Percentage of Appraisal Component Appraisal Estimate Actual/Latest Estimate Bank Govt. CoF. Bank Govt. CoF. Bank Govt. CoF. A. Strengthening 5916.87 1199.83 0.00 7248.75 313.24 97.53 122.5 26.1 0.0 Primary Health Care System A.1. Training of PHC 1981.81 427.74 0.00 1419.64 77.03 0.00 71.6 18.0 0.0 Providers A.2. PHC Development 3432.24 767.58 0.00 5655.54 235.16 97.53 164.8 30.6 0.0 Program A.3. PHC Providers 502.83 4.51 0.00 173.57 1.05 0.00 34.5 23.3 0.0 Guidelines Development Subtotal: Strengthening 5916.87 1199.83 0.00 7248.75 313.24 97.53 122.5 26.1 0.0 PHC Systems B. Strengthening Health 3080.94 775.19 0.00 1334.53 80.89 0.00 43.3 10.4 0.0 Financing System B.1. Establishment of SHA 1549.58 768.53 0.00 625.02 77.38 0.00 40.3 10.1 0.0 B.2. Improvement of BBP 390.80 6.66 0.00 168.43 1.35 0.00 43.1 20.3 0.0 B.3. Improvement of 418.13 0.00 0.00 139.42 2.16 0.00 33.3 0.0 0.0 Provider Payment Methodology B.4. Financial information 722.43 0.00 0.00 401.66 0.00 0.00 55.6 0.0 0.0 System Subtotal: Strengthening 3080.94 775.19 0.00 1334.53 80.89 0.00 43.3 10.4 0.0 Health Financing Systems C. Project Management 1015.56 6.97 150.00 766.39 10.54 0.00 75.5 151.2 0.0 TOTAL PROJECT 10013.38 1981.99 150.00 9349.67 404.67 97.53 93.4 20.4 65.0 COSTS - 28 - Annex 3. Economic Costs and Benefits Quantitative cost-benefit or cost-effectiveness analysis was not undertaken during project preparation or appraisal. The project's focus on primary health care was based on the rational that improved quality and access to primary care would benefit a larger share of the population than a focus on secondary and hospital care; the potential of strengthened PHC to improve access to care by virtue of its proximity to the population and, the pressing need to improve the balance between primary and secondary care in Armenia, in view of improved cost-effectiveness of primary care. These considerations all remain valid to date. In particular, the beneficiary assessment has shown that the population's access to care in target villages improved, out-of-pocket payments reduced, referral and self-referral to specialist care decreased and patient satisfaction increased compared to other villages (see annex 1 outcome indicators). Furthermore, cost-benefit analysis of the proposed expansion of the primary health care component under the recently approved Health System Modernization project, indicated that these interventions have rate of return well beyond the opportunity cost of capital. The direct benefits of the health finance component are more difficult to quantify, although international evidence suggests that the type of systemic reforms supported under the project harbor the potential to substantially increase sectoral efficiency and contain costs. - 29 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, 1 FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 04/07/1996 5 PTL; HEALTH SPECIALIST (1); SOCIAL ASSESSMENT SPECIALIST (1); CONSULTANT (1) 07/27/1996 5 PTL; HEALTH SPECIALIST (1); SOCIAL ASSESSMENT SPECIALIST (1); CONSULTANTS (2); 10/21/1996 7 PTL; HEALTH SPECIALIST (1); PRINCIPAL ECONOMIST (1); SR. ECONOMIST (1); HEALTH ECONOMIST (1); PROJECT IMPLEMENTATION SPECIALIST (1); CONSULTANTS (1); Appraisal/Negotiation 04/24/1997 8 PTL; HEALTH SPECIALIST (1); HUMAN RESOURCES SPECIALIST (1); CONSULTANTS (5) Supervision 03/25/1998 3 HEALTH ECON. & PTL (1); S S OPERATIONS ANALYST (1); PROCUREMENT SPECIALIST (1) 10/19/1999 4 HEALTH ECON AND PTL (1); S S OPERATIONS ANALYST (1); SR. HEALTH SPECIALIST (1); CONS./RM PROJECT OFF. (1) 03/24/2000 4 TEAM LEADER (1); S S OPERATIONS ANALYST (1); PROJECT OFFICER (1); HNP CONSULTANT (1) 10/14/2000 7 PTL/HEALTH SPEC. (1); S S OPERATIONS (1); PROJECT OFFICER (1); OPERATIONS ANALYST (1); HEALTH SECTOR MGR (1); OUTGOING PTL (1); PROCUREMENT SPECIALIST (1) 10/14/2000 4 PTL (1); TTL (1); S S OPERATIONS OFFICER (1); PROCUREMENT (1) - 30 - 05/20/2002 4 PTL (1); TTL (1); HEALTH S S SPECIALIST (1); PROCUREMENT (1) 10/18/2002 6 TTL AND OPERATIONS (1); S S OUTGOING PTL (1); INCOMING PTL (1); PROJECT OFFICER (1); FMS (1); PROCUREMENT (1) 05/25/2001 4 PTL, SR. HEALTH SPEC (1); S S TTL, OPERATIONS OFF (1); LOCAL PROJECT OFFICER (1); PROCUREMENT SPEC (1) 09/23/2003 3 SR. HEALTH SPECIALIST (1); S S SR. HD ECONOMIST (1); OPERATIONS OFFICER (1) ICR 12/01/2003 3 SR. HEALTH SPECIALIST S S (1); SR. HD ECONOMIST (1); OPERATIONS OFFICER (1) 03/06/2004 5 TTL; SR. HEALTH SPEC (2); S S OPERATIONS OFF (1); SR. HD ECONOMIST (1); (b) Staff: Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 55.0* 165.0* Appraisal/Negotiation 15.0* 45.0* Supervision 142.6 274.0 ICR 19.9 39.7 Total 272.5 523.7 * Costs reflected are based on estimated expenses prior to SAP roll-out and include costs of WB staff and long-term consultants at that time. - 31 - Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating Macro policies H SU M N NA Sector Policies H SU M N NA Physical H SU M N NA Financial H SU M N NA Institutional Development H SU M N NA Environmental H SU M N NA Social Poverty Reduction H SU M N NA Gender H SU M N NA Other (Please specify) H SU M N NA Private sector development H SU M N NA Public sector management H SU M N NA Other (Please specify) H SU M N NA - 32 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating Lending HS S U HU Supervision HS S U HU Overall HS S U HU 6.2 Borrower performance Rating Preparation HS S U HU Government implementation performance HS S U HU Implementation agency performance HS S U HU Overall HS S U HU - 33 - Annex 7. List of Supporting Documents 1. Staff Appraisal report No 16475-AM. 2. Bank mission back to office reports and Aide Memoires from 1998-2003. 3. Beneficiary Assessment Report, 2003.. 4. "Review of health financing and provider payment systems in Armenia", consultant report prepared for Ministry of Health of Republic of Armenia, February 2004. 5. Background Paper for Armenia Public Expenditure Review: Health Care. Consultant report, April 2002. 6. Armenia Poverty Reduction Strategy Paper. Yerevan, 2003. 7. Review of the training programs for primary health care providers. Peer review report, WONCA consultant, October 2003. 8. Poverty Targeted Social Assistance and Health: Impact of Eligibility Expansion on Health Care Utilization in Armenia. Edmundo Murrugarra et al. Research Paper, January 2004. 9. Armenia Country Assistance Strategy. May 2004. - 34 - Annex 8. Beneficiary Survey Results A project beneficiary assessment was carried out by independent local consultants in mid-2003. The assessment utilized a combination of qualitative and quantitative survey techniques. Qualitative research was conducted in four regions of Armenia and involved focus group discussions and semi-structured interviews with primary and secondary stakeholders. The quantitative evaluation was carried out in nine regions. It included a household survey (2982 individuals in project target communities and 1632 individuals in control communities), physicians survey, review of ambulatory records, ambulatory patient cards and business plans submitted by trained physicians. This annex summarizes conclusions from this assessment, a more detailed account is available in the project files. Assessment of Project Objectives Overall, stakeholders noted that the project had set the right goals, reflecting the most crucial challenges faced by the health system of Armenia both at project inception and today. Factors affecting project outcomes and implementation Among key factors affecting project implementation and project outcomes, stakeholders mentioned (i) erratic changes in Government policy, particularly on the health finance front; (ii) chronic underfunding and irregular flow of public funds to the health sector, (iii) the lack of quality assurance mechanisms in the health sector; and (iv) lack of medical equipment and supplies at facility level; (v) difficulty of beneficiary communities to adequately maintain renovated facilities and project provided ambulances. Primary Health Care Component The assessment found that the PHC component contributed significantly to improved quality and access to primary health care services in target communities. Informal and out-of-pocket payments for health care were less frequent and lower in the target than in the control communities. The patient referral rate among project trained family physicians was only forty percent of that among those in the control group for first time patients and only one fifth for old patients. Similarly, patient self-referrals to specialists were almost fifty percent lower in target communities. Overall, a larger share of the population in target communities expressed satisfaction with the quality of care provided at their PHC center than in the control group. Similarly, a higher share of the population in target communities confirmed trusting the family physician than in the control villages. The assessment found that beneficiaries widely agreed that family medicine could be considered as having established itself as a credible discipline in Armenia with the help of the project and that the latter could overall be considered as having achieved its goals. Wide recognition was given to the successful establishment of family medicine chairs at three training institutions with the project's support. There was also wide agreement that family medicine training had greatly benefited from materials developed and supplied under the project, including the clinical guidelines, specialized literature and audio-visual materials. Trainees mentioned that the main shortcoming of the family medicine retraining program were the relatively short to duration of training, insufficient clinical training in a family medicine setting and the limited number of patients seen per trainee when there was clinical training. It was suggested that the training could be improved by carrying it out in modular fashion with theoretical training followed by clinical training in a family medicine setting during each module. Training beneficiaries also pointed out that regular follow up training through a continuous education system would be beneficial. It was also - 35 - mentioned that trained staff are some times unable to find jobs, which negatively affects the reputation of the training courses. Many stakeholders noted that the inclusion of trained family physicians in the PHC system is more difficult in urban than rural areas because of the presence of narrow specialists in urban policlinics. The problem of integrating family medicine physicians into policlinics was also seen as being related to the system of remuneration for policlinics. While the work of family physicians is paid by SHA mostly on a capitation basis, work of specialists is carried out on a fee for service basis, both under SHA contracts and for out of pocket payments. Therefore, specialists are seen as the main source of additional income for policlinics and policlinic managers are keen to assure that specialists are on board and have a sufficient number of patients to treat. It was also considered important that trained family doctors were able to work in rehabilitated and reequipped ambulatories in the Marzes, i.e. they were granted the necessary conditions for practicing the knowledge and skills they had acquired. Clinical guidelines for family medicine doctors and nurses developed under the project were considered to be an effective tool to improve management of patients' health problems, although it was mentioned that guidelines should be more widely distributed to all practitioners. Many recommended that the guidelines be periodically updated, provide modern treatment methods, provide more detailed explanations of treatment methods, recommend pharmaceuticals ,which are indeed available in the local pharmacies and that guidelines be published as a book. Health care providers in target communities were overall highly satisfied with the outputs of the project and considered that the project had overall contributed to better life in the village by improving quality and affordability of care. Similarly the heads in a number of target communities mentioned that the reach of the project was beyond their communities, with the population from neighboring communities becoming more aware and interested in health care issues and wanting to participate in project activities to improve the health care provider conditions in their communities. Community leaders emphasized the importance of mobilizing community members to improve the impact of the project. Health Finance Reforms Chronic underfunding of the sector and irregular resource flows were generally seen as the main factors which had prevented the health finance reforms supported under the project from having their full effect. Many health care providers associated the SHA directly with the poor financing situation in the sector and thus had an overall negative view of the agency and reforms associated with its establishment. The assessment found that there had been a wide spread believe that establishment of SHA and introduction of new provider payment systems would bring about an improvement in the financial situation of providers, ignoring the fact that the latter required an increase in public sector funding for health care and substantial reduction of overcapacity. The beneficiary assessment concludes that the medical community in general did not understand the role of SHA sufficiently and unfairly assigned the chronic funding shortage and provider level funding shortages related to sectoral over capacity to the agency and changes in payment mechanisms introduced with its establishment, rather than to unsupportive Government policies. The assessment also found that many beneficiaries criticized the SHA for not having introduced clear quality control mechanisms, though the assessment concludes that quality control is clearly an issue which reaches beyond the scope of SHA and requires the active involvement of MOH and associated bodies. The assessment points to an opinion expressed by many beneficiaries that the limited impact of health finance reforms was due to the fact that they only covered a minority of a provider's overall income sources and were thus not able to substantially influence provider behavior. This shortcoming not withstanding, the assessment concludes that the introduction of global budgets for hospital and specialist care and of capitation payments for PHC constituted a significant achievement as it put a halt to previously chronic - 36 - arrears accumulation to providers and enticed provider managers to improve internal financial management. Provider managers generally positively assessed the new reporting system introduced by SHA and found that the reporting system also helped them gain better oversight of provider activities and resource use. The assessment concludes that while the work carried out by the BBP working group had its merits on technical grounds and heightened the awareness of the need to limit public funding to cost effective interventions of high priority in Armenia, the overall definition of the BBP remains predominantly a process driven by political influence of various interest groups in the medical community. The assessment found that there is recognition among policy makers and SHA staff that the BBP remains underfunded, but that there are continuous calls from providers, the population and politicians alike to further expand the contents of the BBP. Some PHC providers also reported that they were not able to effectively limit free service provision to services included under the BBP, as local politicians demanded that they provide all services free to the local population irrespective of providers' contracts with SHA. Performance of the PCU Overall, stakeholders spoke positively of collaboration with the HPIU. SHA representatives, leaders of target hamaynks, and directors of HCPs that dealt with the project, as well as trained staff mentioned that the PIU was always there to respond to their needs and to organize field work efficiently. - 37 - Annex 9. Stakeholder Workshop Results Two beneficiary assessment workshops were held in Yerevan to assess achievements and outstanding issues on the health finance and primary health care development fronts. The workshop on primary health care took place December 9-10 2003, and was attended by a wide range of stakeholders, including representatives of MoH, SHA, medical faculty, health care providers, regional health directorates, local Governments, donors and NGOs. The workshop on health finance reform was held on January 29, 2004 and attended by representatives of MoH, SHA, MOF, hospitals, primary care providers, local governments, medical faculty, donors and consultants which had carried out an independent assessment of the status of the health finance reforms in Albania. A list of participants at both workshops is presented in appendix 1 to this annex. Health Finance Reform Workshop: The workshop focused on a discussion of the overall status of the health finance system, effects of reforms undertaken so far and in what direction the reforms should go in the future. Folded into this discussion was the outcome of the IDA financed project that supported the key health sector financing reforms. The workshop was guided by the the findings of independent international consultants who had been commission to undertake a thorough analysis of the current status of the health finance system. Presentations by consultants were followed by discussions among stakeholders on various aspects of the status of the health finance and provider payment system. Key conclusions were: l The health finance and provider payment system should not be seen in isolation from the overall organization of the health care delivery system; financing mechanisms and provider payment reforms alone will not guarantee performance and quality of care. l The current provider payment system includes some irrational, conflicting incentives, particularly at the polyclinic level. These are exacerbated by the continued coexistence of primary care providers/family physicians and narrow specialists within the same setting and the fact that specialist outpatient services are not integrated with hospital services. l Provider payment reforms must be supported by service delivery reforms to have their desired effects, including rationalization of the provider network. l Provide payment reforms must be supported by actions to strengthen provider management skills and to improve reporting and accounting systems. The latter are critical to allow for more accurate cost estimates of service provision. l SHA and its payment mechanisms can not be expected to have a significant impact on provider behavior, particularly at the hospital level, where revenues from state orders often constitute less than have of all provider income. l Public sector funding has been poor and erratic for most of the time since health finance reforms have been initiated, which has severely limited the effectiveness of reforms. l Provider payments are not linked to performance targets and do not include any performance rewards. l The frequently changing scope and contents of the BBP has led to confusion among providers and patients about what services are supposed to be available for free to what population groups and provided room for abuse; insecurity of what costs for services will be may also have led to decreases in - 38 - demand for health care, particularly among the lower income groups which in principle are supposed to be provided with relatively broad BBP coverage. l Armenia's current health finance system is focused on available resources rather than patient oriented. l Further improvements and adjustments to the health finance system should be preceded by a careful study of sources and uses of funds in the health sector and detailed study of the cost structures in the system. l Patients currently pay close to half of all health care costs through official and unofficial payments. These payments should be officialized through introduction of broader co-payment schemes with differential co-payment rates for various target groups. l Turning informal payments into formal payments improves patient rights and patients'voice, but it requires strong political will and leadership as well as aggressive information campaigns to ensure the population is aware of the change and the reasons behind it. l As medium term objective, as public funding for health care will increase (as per PRSP/MTEF) consideration should be given to bring out of pocket payments as a share of total health care spending down, while allowing public funds to cover an increasingly larger share of demand so as to assure greater access to services. l Further payment reforms should be accompanied by other mechanisms to improve the performance of the health system, including efforts to improve quality of care (quality assurance mechanisms; accreditation and licensing); other supportive measures that could help improve the system's performance might be strict introduction of PHC providers as gatekeepers and higher fees paid by those who circumvent PHC. PHC Workshop: The workshop reviewed the current status of PHC development in Armenia, based on the experience of activities implemented by the Government with support from the IDA financed project and other donors. Presentations by officials from the Ministry of Health, State Health Agency, PHC training institutions, Marzes, PHC projects supported by other donors and consultants were followed by discussions among stakeholders on the lessons and future of PHC reforms in Armenia. Key conclusions were the following: l Based on the successful implementation of its first PHC strategy, an updated PHC development strategy was approved by the Government in 2003. The new strategy supports the scale up and completion the PHC reforms, based on the family medicine approach validated by the implementation of the IDA supported health project. l Retrained and newly trained family doctors started developing their identity as a professional group. They have established a professional association to represent their interest. Government has recognized family medicine as a specialty with the same status as other medical specialties. l Government commitment exists for increasing the share of public expenditures for PHC and SHA has proposals for changing payment mechanisms for primary care. However, further technical work and changes of regulations are needed to properly align incentives of the payment system for family doctors with their new range of skills. The payment system in place currently for polyclinics, where both primary care providers and specialists work, is distorted: the component of capitation payments for - 39 - primary care subsidizes costs of specialist care, whereas direct payments for specialists provide incentives for expanding their services. l Representatives of local communities where subproject were implemented expressed their satisfaction with the involvement in carrying out the project and the willingness to continue to contribute to decision-making and support for development of PHC services. l Further improvements of the legal and regulatory framework are needed to ensure that the family medicine approach can be effectively put in practice by newly trained PHC staff. The legal status of the family medicine needs to be better defined by codifying in law the Armenian Family Medicine Statement, allowing for autonomous management and direct contracting with SHA for family medicine practices and relaxing the current rigid patient assignments by catchment areas. New organizational arrangements would have to be implemented, integrating into hospitals the specialist services currently provided in polyclinics and setting up independent family medicine providers. l Although different views and proposals exist regarding improvements of the training curriculum for family medicine training, there is agreement on the need to expand clinical training in family medicine settings, supplementing the already well developed class room and hospital based components of the training. Therefore, additional clinical training capacity in PHC setting needs to be added to the existing facilities, requiring investments in infrastructure and recruitment and training of more family doctors to supervise practical training. The new Unified Family Medicine Curriculum, with a duration of 18 months, although excellent in content, was deemed to be too long and sophisticated to be used for the retraining of physicians; it was proposed to continue using the 11 months curriculum during the transition period, covering the same subjects as in the unified curriculum, and a division of about one third theory and two thirds practice. The unified curriculum of 18 months could be introduced in the coming years for the residency programmes. - 40 - Appendix 1 List of Participants at Health Finance Workshop 1 Darbinyan Hayk Deputy Ministers 2 Hakobyan Tatul MOH 3 Arman Hovhannesyan Senior Specialist, Economic Department 4 Poghosyan Vahan Head of Health Department 5 Juzbashyan Ruzanna Head of PHC division 6 Qrmoyan Suren Legal Adviser of the Minister of Health 7 Ter-Grigoryan Ara Director of SHA 8 Antonyan Karlen Deputy Director of SHA SHA Head of Procurement and Information 9 Kharazyan Samvel Department 10 Kolozyan Khachik Head of Yerevan SHA branch 11 Arsen Davtyan Head of Financial Department 12 Nane Grigoryan Senior Specialist on Health issues MOF 13 Naira Davtyan Senior Specialist Health issues 14 Gayane Davtyan Senior Specialist Merger 1 Chief accountant of "Surb Grigor Lusavorich" 15 Shahbatyan Leyla hospital merger Merger 3 16 Stamboltsyan Gagik Director of hospital merger Head of outpatient service at the University Merger 4 17 Suqiasyan Robert Hospital merger 18 Shaqaryan Armen Prorector of State Medical University NIH 19 Hovhannesyan Samvel Head of FM Chair Polyclinic N17 20 Sahakyan Gagik Director of the Polyclinic #17 WB Armenian Office 21 Hayrapetyan Susanna Senior Health Specialist 22 Khachatryan Sergey HPIU Director 23 Elibekyan Yervand PIU 24 Harutunyan Hasmik 25 Aydinyan Lusine 26 Petrosyan Lilit 27 Tsaturyan Saro Director of Diagnostic Center Other mergers/hospitals 28 Seinyan Sergey Deputy Director of Oncological Scientific Center 29 Norikyan Taguhy Chief Accountant of Orthopeadic Hospital 30 Torosyan Samvel Director of Psychiatric Hospital 31 Peter Droog 32 Joyce Smith Conseil Sante 33 Bernardo Ramires Consultants 34 Raimond Bentz 35 Cristopher Lamiere USAID, PADCO 36 Tatyana Makarova 37 Nancy Fitch - 41 - List of Participants at PHC Workshop Ministry of Health Haik Darbinyan Deputy Minister Tatul Hakobyan Deputy Minister Suren Krmoyan Legal Adviser to the Minister Ruslana Gevorgyan Adviser to the Minister Gagik Sayadyan Head of the Staff Vahan Poghosyan Head of the Health care Department Ruzanna Yuzbashyan Head of the PHC sub-department Lida Poghosyan S. Ghasabyan Knar Ghonyan Nora Pahlevanyan Lida Poghosyan Ella Safaryan Ara Babloyan Former Minister of Health Ara Ter - Grigoryan Head of SHA State Medical University Vilen Hakobyan The Rector Marina Ohanyan Post-graduate student (Ordinator) 3 other ordinators National Institute of Health Samvel Hovhannisyan Head of the Family Medicine faculty L. Hovhannisyan Ordinator N. Asatryan Ordinator V. Mantasheva Basic Medical College Donara Hakobyan Head of the Family Medicine faculty Tereza Hakobyan Varduhi Arzumanyan Naira Nersisyan - 42 - Yerevan Policlinics Gagik Sahakyan Director of PC 17 Dz. Papoyan Physician from PC 17 A. Divazyan Physician from PC 17 S. Badalyan Physician from PC 17 P. Ghukanjyan Physician from PC 17 Sargsyan Physician from PC 17 Armen Kerobyan ASTP representative in PC 17 Vahram Karapetyan Director of PC 2 Sergey Hairapetyan Director of PC 10 Samvel Sargsyan Director of PC 19 Tigran Khachatryan Director of Medical Union N 2 Anahit Ghazaryan Ararat Marz A. Chobanyan Director of the ambulatory of Taperakan hamaynk M. Gharibyan Family physician of Taperakan ambulatory M. Manaseryan Head of Taperakan hamaynk Ts. Alexanyan Director of the ambulatory of Ararat hamaynk A. Begijanyan Director of the ambulatory of Shahumyan hamaynk Armavir Marz L. Muradyan Head of the Healthcare department of Armavir Marz local governing body (Marzpetaran) Sh. Tovmasyan Director of the ambulatory of Haytagh hamaynk S. Vardanyan Head of Haytagh hamaynk S. Hakobyan Director of Armavir policlinic Aragatsotn Marz Silva Tadevosyan Director of ambulatory of Agarak hamaynk Gegharkunik Marz G. Davtyan Head of the Healthcare department of Gegharkunik Marzpetaran D. Grigoryan Director of ambulatory of Noratus hamaynk Kotayk Marz S. Muradyan Head of the Healthcare department of Kotayk Marzpetaran A. Jumayan Director of ambulatory of Balahovit hamaynk K. Grigoryan Head of Balahovit hamaynk - 43 - Tavush Marz L. Matinyan Director of ambulatory of Gandzakar hamaynk Lori Marz R. Dilbaryan Head of the Healthcare department of Lori Marzpetaran A. Gharajyan Director of ambulatory of Vardablur hamaynk Syunik Marz S. Ghahramanyan Director of Kapan policlinic L. Harutiunyan Director of Goris policlinic Shirak Marz R. Khachatryan Head of the Healthcare department of Shirak Marzpetaran M. Danielyan Director of ambulatory of Jajur hamaynk S. Petrosyan Director of PC N 3 of Gyumri Susanna Sargsyan Physician of PC N 3 of Gyumri Sargis Karapetyan Physician of PC N 3 of Gyumri Karine Santrosyan Physician of PC N 3 of Gyumri Lusine Antonyan Physician of PC N 3 of Gyumri Vayots Dzor Marz Karapetyan Director of Eghegnadzor policlinic M. Movsisyan Head of Malishka hamaynk A. Aslikyan NGOs Armenian Relief and Development (IRD) Mariam Sianozova Vladimir Kogan Irina Dokhikyan OXFAM Margarita HakobyanProgram representative Hasmik KocharyanHealth program manager Karen Arakelyan United Methodist Committee (UMCOR) Araks HovhannisyanHealth program manager USAID, AIHA, PADCO Nancy Fitch Ruzanna Stepanyan Gayane Gharagebakyan Anna Grigoryan Ruben Jamalyan Vahe Isahakyan Mher Manukyan - 44 - World Bank Toomas Palu AM Health Team Leader, Sr. Health Specialist Monika Huppi Sr. Human Development Economist Silviu Radulescu Sr. Health Specialist Tamar Gotzadze Projects Officer Susanna Hairapetyan Sr. Health Specialist, Project Officer in Yerevan office WHO Elizabeth Danielyan WHO Representative CSIH Michal Porter John Millar Conseil Sante Peter Droog Team Leader Cristophe Lemiere Hospital financial planning expert Joyce Smith Human Resources expert HERA Marc Reveillon Health sector financing expert Kees SchaapveldHealth consultant "Development Programms"Ltd Saro Tsaturyan Children's Univesity hospital N 1 Hayryan Marina Poghosyan Karmella HPIU Sergey Khachatryan Director of HPIU Yervand Elibekyan PHCDP Manager Lusine Aydinyan Financial Component Coordinator Nune Arzakanyan Evaluation Specialist Aida Khachatryan PHCDP support specialist Garik Aghabekyan Head of Training Component Gayane Muradyan IT specialist Karine Simonyan PR specialist - 45 - - 46 -