95083 VERIFICATION OF PERFORMANCE IN RESULTS-BASED FINANCING PROGRAMS: THE CASE OF PLAN NACER IN ARGENTINA DISCUSSION PAPER NOVEMBER 2014 Alfredo Perazzo Erik Josephson VERIFICATION OF PERFORMANCE IN RESULTS-BASED FINANCING PROGRAMS The Case of Plan Nacer in Argentina Alfredo Perazzo and Erik Josephson November 2014 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Global Practice. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For information regarding the HNP Discussion Paper Series, please contact the Editor, Martin Lutalo at mlutalo@worldbank.org or Erika Yanick at eyanick@worldbank.org. © 2014 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper VERIFICATION OF PERFORMANCE IN RESULTS-BASED FINANCING PROGRAMS: The Case of Plan Nacer in Argentina Alfredo Perazzoa Erik Josephsonb a GHNDR, The World Bank Group, Washington, DC, USA b GHNDR, The World Bank Group, Washington, DC, USA Paper prepared for Discussion Paper Series on Verification of Performance in RBF The World Bank Group, Washington, DC, USA November 2014 Abstract: Verification in results-based financing (RBF) mechanisms is one of the key differentiators between it and related health financing structures such as social health insurance. Verifying that providers have achieved reported performance in RBF mechanisms is considered a crucial part of program implementation and key to maintaining trust through transparency, as well as the viability of the mechanism. Verification is however a process which has thus far been little studied. Information on the methodologies used in different settings (including frequency and sampling methodology), the effectiveness of the verification process, the direct and tangential effects, and the cost is scarce. Plan Nacer employs one of the largest RBF mechanisms in the world and is therefore an excellent case study for the role, methodology and effects of the verification process. This study will give the background to Plan Nacer, detail the major characteristics of the verification process and draw lessons on the process which can inform the design of verification in RBF mechanisms in other countries. Keywords: Plan Nacer, Verification, Results-Based Financing Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: Petra Vergeer, The World Bank Group, 1818 H Street, NW Washington, DC 20433, 202-473-7181, pvergeer@worldbank.org 3 Table of Contents ACKNOWLEDGMENTS.................................................................................................................. 6 PREFACE ........................................................................................................................................ 7 PURPOSE AND OBJECTIVES OF THE STUDY ........................................................................... 8 METHODOLOGY ............................................................................................................................ 8 COUNTRY CONTEXT ..................................................................................................................... 9 OVERVIEW OF PLAN NACER ......................................................................................................... 10 INTERNAL ORGANIZATION OF PLAN NACER LINKED TO SUPERVISION AND VERIFICATION.............................................................................................................................. 16 REGULATIONS GOVERNING PLAN NACER ...................................................................................... 16 ORGANIZATION OF PLAN NACER ................................................................................................... 18 INFORMATION SYSTEM ................................................................................................................. 18 ROLES OF ACTORS INVOLVED IN VERIFICATION ............................................................................. 19 MAJOR CHARACTERISTICS OF THE VERIFICATION METHOD ............................................. 19 W HO DOES THE VERIFICATION? ................................................................................................... 19 THE SCOPE OF W ORK OF THE THIRD PARTY VERIFIER .................................................................. 20 HOW IS VERIFICATION DONE? ...................................................................................................... 22 Verification of Beneficiary Enrollment .................................................................................... 22 Counter-Verification of Beneficiary Enrollment ...................................................................... 23 Tracer Verification .................................................................................................................. 23 Verification of the Use of Provincial Funds to Purchase Services ......................................... 25 Other General Verification ..................................................................................................... 25 Operational Monitoring....................................................................................................... 25 Evaluation of the Work of the Third Party Verifier and Reports Approval .......................... 26 VERIFICATION RESULTS ............................................................................................................ 27 VERIFICATION RESULTS ON BENEFICIARY ENROLLMENT ................................................................ 27 VERIFICATION RESULTS ON TRACERS ........................................................................................... 28 ADJUSTMENTS AND FINES BASED ON VERIFICATION ...................................................................... 30 SAMPLING ................................................................................................................................... 31 DISSEMINATION PROCESS ............................................................................................................ 32 COST OF THE THIRD PARTY VERIFIER .......................... ERROR! BOOKMARK NOT DEFINED. SUMMARY AND CONCLUSIONS ................................................................................................ 36 REFERENCES .............................................................................................................................. 40 ANNEX 1: ORGANIZATIONAL STRUCTURE OF THE UEC IN 2011 ......................................... 41 ANNEX 2: VERIFICATION TASKS OF THE THIRD PARTY VERIFIER RELATED TO BENEFICIARY ENROLLMENT .................................................................................................... 42 ANNEX 3: THIRD PARTY VERIFIER OPERATIONAL VERIFICATION DETAILS...................... 43 4 Tables and Figures Table 1 Roles and institutional framework of Plan Nacer .............................................................. 12 Table 2 Tracer Definitions.............................................................................................................. 13 Table 3 Classification of Regulations Implemented by the UEC ................................................... 17 Table 4 Responsibilities in the UEC Departments ......................................................................... 18 Table 5 Main Actors' Responsibilities in the Verification Process ................................................. 19 Table 6 Third Party Verifier's TOR ................................................................................................. 20 Table 7 Records Processed and Rejected- Plan Nacer 2004 2012 .............................................. 27 Table 8 Error Rates Higher than 20 and 40 Percent by Indicator, 2008-2012............................... 29 Table 9 Percentage of Cases in Which One Province Moved Down a Tier for One Tracer Following Verification ............................................................................................................. 29 Table 10 Adjustments and Fines Related to Payment Transfers from 2005-2012........................ 31 Table 11 Contract in US$ for Different Phases and Portions of the Project .................................. 33 Table 12 Third Party Verification Team Composition .................................................................... 35 Table 13 Cost of Third Party Verifier ............................................................................................. 36 Figure 1 Measurement of Tracer Target Achievement from 2004-2008 ....................................... 14 Figure 2 Measurement of Tracer Target Achievement After 2008 ................................................ 15 Figure 3 Third Party Verifier's Tracer Verification Process ............................................................ 24 Figure 4 Percentage of Declared Records Rejected by the Third Party Verifier ........................... 28 5 ACKNOWLEDGMENTS The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. This paper has benefited greatly from assistance from Programa Sumar of the Ministry of Health, Argentina. The report has also greatly benefited from comments provided by the Ministry of Health officials including Martin Sabignoso, Humberto Silva, Martin Piazza, Ricardo Izquierdo, Ana Sala, Hernan Diaz Vera, Alvaro Ocariz and their teams. The paper has also benefited from comments by World Bank staff including Daniela Romero, Vanina Camporeale, Paula Giovagnoli, David Groisman and Petra Vergeer. Finally, the authors gratefully acknowledge financial support from the World Bank Health Results Innovation Trust Fund (HRITF). The content and conclusions expressed in this paper are those of the authors themselves and do not necessarily represent the views of the World Bank, Argentine National Ministry of Health, or the funders of the research. All errors and omissions are the responsibility of the authors alone. 6 PREFACE Argentina is located in the south eastern part of South America. It is a federal country with 23 provinces next to the capital Buenos Aires1. This middle-income country and its approximately 41.6 million inhabitants faced a significant economic crisis in early 2000s. Health sector reforms were initiated to ensure access to public health services for the poor while at the same time recognizing the need to improve the quality of care. Among other policies, Argentina used a results-based financing (RBF) approach called ‘Plan Nacer’ to accomplish this. The program was piloted in nine of the poorest provinces in the country and subsequently expanded nationwide. Through Plan Nacer provinces are incentivized to enroll uninsured pregnant women and children under 6 years of age to ensure they have access to basic health services. The Ministry of Health at federal level decides on the benefit package and pays the provinces 60 percent of a capitation fee for each beneficiary enrolled in Plan Nacer. The remaining 40 percent is paid based on achievement of ten health indicator targets (called ‘tracers’) that are agreed upon with the provinces. This results-based payment mechanism is called Plan Nacer. More recently the government has expanded the RBF approach to include women and adolescents and a wider package of service called Program Sumar. In both schemes, the Provincial Health Insurance uses RBF payment and other funds to reimburse the providers on a fee-for-service basis for the agreed upon services provided to the target population. This study aims to describe the methods used to verify the results in Plan Nacer, to present the results, and discuss some of the challenges and lessons learned. There are four types of verification mechanisms in the scheme:  Verification of beneficiary enrollment carried out by the Central Executive Unit (called ‘UEC’) of Plan Nacer. On a monthly basis the enrollment records are checked for duplications and compared against databases of other insurance schemes.  Counter-verification of the beneficiaries’ enrollment is performed by a third party through checking the consistency and source of the information every 2 months.  The third party uses this same process to help determine the provinc es’ performance against the tracer indicators. In addition, a sample of clients is verified at the health facility level by a review of the clinical records and checking of, for example, the beneficiary signature on the enrollment form. The third party verifier reports these results every four months to the UEC.  The third party verifies the use of the funds received by the provinces through Plan Nacer to ensure it is used for the purchasing of services covered by the program. This is checked by the third party at facility level, usually at the same facilities mentioned in the previous step. The following results of verification were identified by this study:  In the first phase of Plan Nacer (9 provinces) there was a sharp decline from 20 percent of beneficiary enrollment records rejected by the third party in the first year to 2.7 percent in year two and ranging between 0.1-1 percent in subsequent years. The subsequent nationwide roll-out started out with lower rejection rates ranging between 0.4-0.7 percent  The verification of tracers by the third party also saw a decline in rejection rate from 25 percent of the records declared by the provinces in 2005 to a slower progression to approximately 5 percent of the declared records rejected in 2012. These verification findings are used to make adjustments to the RBF payments to the provinces and when necessary impose fines in line with the operational manual of the program. Between 1 The capital Buenos Aires has a similar status as any of the Provinces in this federal system. 7 2004 and 2012 this represented 5.9 percent of the gross transfers with the majority, $8.3million, due to adjustments to tracer payments and $2.75million for beneficiary enrollment adjustments. Seeing this in light of the cost of the third party verifier, at a value of $21.6million during this same period, one could say that more than half of the costs of the third party verifier are recuperated through fines and adjustments to the RBF payments. Of course it is not known what the level of error in reporting would have been without the involvement of the third party and thus the total amounts saved because of the verification. The verifier costs are influenced by the team composition (both number and qualification) as well as the sample size with corresponding field visits. There has been tension between the objectives of the third party and those of the UEC. It is in the interest of the third party verifier to limit the size and distribution (mostly large health facilities) of the sample selected but to be sufficient to form their independent opinion while it is in the interest of the UEC to have as many facilities as possible verified to create an environment of control. A larger sample size and inclusion of smaller health facilities would lead to additional cost; consideration may also be given to whether other actors or tools can play this role. The role of the provinces versus the role of the UEC at national level in this verification at health facility level should also be taken into account. The third party verifier has had a positive influence on the Plan Nacer program by ensuring that reporting errors are corrected and RBF payments adjusted accordingly. Moreover, it has functioned as a mediator between the national government and the provinces. While it is formally not mandatory for the government to follow the opinion of the third party, in practice 99 percent of the recommendations of the third party are implemented. This use of a third party to help govern the relationships is especially suitable given the federal nature of Argentina. PURPOSE AND OBJECTIVES OF THE STUDY Verification in results-based financing (RBF) mechanisms is one of the key differentiators between it and related health financing structures such as social health insurance. Verifying that providers have achieved reported performance in RBF mechanisms is considered a crucial part of program implementation and key to maintaining trust through transparency, as well as the viability of the mechanism. Verification is however a process which has thus far been little studied. Information on the methodologies used in different settings (including frequency and sampling methodology), the effectiveness of the verification process, the direct and tangential effects, and the cost is scarce. Plan Nacer employs one of the largest RBF mechanisms in the world and is therefore an excellent case study for the role, methodology and effects of the verification process. This study will give the background to Plan Nacer, detail the major characteristics of the verification process and draw lessons on the process which can inform the design of verification in RBF mechanisms in other countries. METHODOLOGY To complete this case study, project documents were reviewed and meetings with key actors were held. The review of project documents provided information on Plan Nacer’s organizational structure, regulatory framework, and its control and verification methods. The primary focus of the meetings with key actors was to discuss the organizational features of Plan Nacer that are related to its supervision and verification. This case study is part of a broader analysis of multiple country case examples to expand knowledge about the verification process and practices to address the immediate design and 8 implementation needs of RBF programs. The other country mechanisms studied in the series are Afghanistan, Burundi, Panama, Rwanda and the United Kingdom. A cross-case analysis will draw together the lessons from the six cases to identify trends, commonalities, differences and provide recommendations for the design of new verification processes. Lastly the cross-case analysis will provide suggestions for further analysis and research. All the case studies in the cross-country analysis follow a similar structure for comparative purposes. Therefore, this case study is organized into the following sections:  Country Context  Overview of Plan Nacer  Internal Organization of Plan Nacer Linked to Supervision  Major Characteristics of the Verification Method  Verification Results  Costs of the Third-Party Verifier  Summary and Conclusion COUNTRY CONTEXT In 2001-2002 Argentina faced one of the biggest socio-economic and political crises in its history. Gross Domestic Product (GDP) decreased by almost 11 percent in 2002 and by 18.9 percent between1998 and 2002. A devaluation of the peso of 300 percent resulted in a decrease in per capita income from US$8000 to US$2695, and poverty increased by 20 percent in 2002 (World Bank, 2004). Despite the gravity of the situation, it was an opportunity for the Government to take steps towards Health Sector Reform. Historically, levels of health expenditure compared with GDP were higher in Argentina than in most countries in the 1990s, but health outcomes did not reflect that. In the early 2000s when Plan Nacer was being designed, maternal and child health was particularly poor in comparison to other middle income countries in the region such as Chile, Costa Rica, and Uruguay. The infant mortality rate had slowed or stagnated at around 17 deaths per 1000 live births after 20 years of a descending trend, and even increased in the poorest Provinces, especially in the Northwest (NOA) and Northeast (NEA) regions2. By contrast, infant mortality in Costa Rica (around 8 deaths per 1000 live births), Chile (around 10) and Uruguay (around 14), was lower. Similarly, the Maternal Mortality Ratio, which had been declining, appeared to increase, trending from 52 maternal deaths per 100,000 live births in 1990 to 44 in 1995, 35 in 2000, and 39 in 2005. Cost Rica, Chile, and Uruguay generally saw rates continue to decline3. The health sector reforms were aimed at improving the availability and effectiveness of public health services to improve the health of the population, especially the poorest. The reforms included: a) broadening the Primary Healthcare Strategy, b) adopting insurance-based schemes to improve access, as well as quantity and quality of services, c) designing incentive and results- based financing mechanisms, d) expanding decentralization in decision-making for expenditure allocation. The new public policy of directly tying results to financing was a substantial modification of common practices used by the governments and financial entities. This required transitioning from a traditional and bureaucratic model of public sector management focused on the purchase of inputs, central intervention, and formal completion, to a new approach with an emphasis on incentives, decentralized responsibility and a focus on impact and results, user satisfaction, transparency and social monitoring. 2NOA refers to North West Region composed of the following Provinces: Salta, Jujuy, Tucumán, Catamarca and Santiago del Estero. NEA refers to North East Region composed of the Provinces of Misiones, Corrientes, Formosa and Chaco. 3Source: Argentina National Ministry of Health 9 OVERVIEW OF PLAN NACER As part of a larger program implemented by the Ministry of Health, a health insurance plan targeting mothers and children, specifically pregnant women and children under six years of age without an explicit insurance coverage, was created. That program, called Plan Nacer, was designed between 2003 and 2004. The objective of Plan Nacer was to ensure that the eligible population had access to a basic package of services. The package was selected based on its importance in improving population health. To this end, Plan Nacer emphasized preventive actions. Plan Nacer was partially funded by the World Bank 4, supplementing existing national and provincial government funds, and included a set of incentives for achieving explicitly defined results. The program aimed to contribute to the reduction of maternal and child mortality and morbidity, and thus to achieve the maternal and child health-related Millennium Development Goals by 20155. One of the main purposes of Plan Nacer was to contribute to the creation of provincial-level insurance programs for mothers and children and, once it was fully developed, to a further expansion to other populations and other service packages. So far, Plan Nacer has had three stages: During the first phase, the project was piloted in nine Provinces of the Northwest (NOA) and Northeast (NEA) regions, chosen for their particularly sensitive poverty situation. The second phase extended the same scheme to the rest of Provinces of the country. The third stage, which is just beginning to be implemented in all Provinces, expands both the eligible population and the service package on the same conceptual basis. In the latest phase, the program is named Programa SUMAR or SUMAR Program. This report focuses on the first two phases of Plan Nacer. One important factor is that Argentina has a federal political system: there are 24 jurisdictions that can participate in the Program, all with the same institutional status (23 Provinces and Buenos Aires City). For simplicity in this study, all entities will be called Provinces. To participate in the program and access its benefits, Provinces must sign a five year agreement with the national government called an Umbrella Agreement. Several actors are involved in the execution of the Program: the Ministry of Health (National Government), provincial governments, and health providers. The structure is based on clearly defined functions and follows Argentina’s federal structure. The Plan offers participating Provinces great freedom in establishing their own guidelines to use these reform tools to best meet their specific needs, for instance, planning for enrollment, developing internal information systems, 4 Plan Nacer was financed by two sections of an Adaptable Program Loan, one of US$135,8 million (project unique identifier P071025, approved on April 15, 2004) and targeting the nine poorest Provinces in the north; the other of US$300 million (project unique identifier P095515, approved on November 2, 2006) is scaling up the project to cover the entire country. At the time of writing this study, a third operation is in the initial processes and will receive US$400 million in financing (project unique identifier P106735, approved on April 28, 2011), and it will include an extension of the project to a broader segment of the population (not only mothers and young children but also women and adolescents) and a larger service coverage. For this last stage, the name of the Program was changed to “Programa SUMAR”. 5In2010 the scope of Plan Nacer was extended to cover the treatment of congenital heart disease. This is coverage for events of low incidence but with a high cost of treatment, and in that it resembles catastrophic insurance. For that, a part of the per capita sum is put aside to cover these services that are provided centrally by a specialized team. The payment order is made through the Provincial Insurance Management Unit (UGSP) and the payment is made directly by the Unidad de Financiamiento Internacional , International Financing Unit (UFI). 10 designing and implementing incentives for personnel, pricing each service in the Package, defining administrative and monitoring processes. The role of the national government is to coordinate and regulate the project, to prioritize the benefits of the Health Care Plan as well as the tracers, and to supervise project implementation; the national government transfers resources to the Provinces, based on enrollment and health results, and finances equipment and technical assistance. The Provinces are responsible for (i) creating a Provincial-Maternal Child Health Insurance (SMIP) through which the benefits of Plan Nacer are accessed, (ii) registering beneficiaries and (iii) updating the beneficiary registry. They are also responsible for purchasing the necessary health services and are accountable for the use of funds and the supervision of health facilities. The health facilities are responsible for the provision of contracted services in accordance with established quality standards. Health facilities receive a payment for each service according to the prices established by the Province. The funds can be managed by the health facility (if it has an appropriate administrative structure) or through a third party administrator (usually another health facility). Regardless, the health facility that owns the funds decides how to use or allocate the money. On top of that, Provinces can define eligible expenses (for instance some Provinces do not allow expenses for certain items such as fuel, travel or hiring of staff). The allocation of funds is to be done according to provincial rules that fulfill certain conditions established by the program. Each Province can decide to adopt (or not) a system of incentives to be paid to provider staff. If they adopt this system, up to 50 percent of the funds may be used to incentivize individuals, while the rest has to be spent on medical supplies, equipment and investments as determined by the expenditure director of the facility. In the first stage of the project, only four of nine Provinces introduced systematic incentives, with considerable variability among them (Argentina National Ministry of Health, Plan Nacer 2011). In the second phase of the program, most Provinces did not implement staff incentives. There are two sets of relationships among the National Government, provincial governments, and health providers which are regulated by agreements (see table 1). On the one hand, the relationship between the National Government and the Provinces is regulated by the Umbrella Agreement which establishes the rights and obligations of each party. It also includes the mechanism through which funds are transferred from the National Government to the Provinces. On the other hand, the relationship between Provinces and providers is regulated by a Management Agreement which governs the billing and payment for the provision of health services to beneficiaries. These payments are made on a fee for service basis. 11 Table 1 Roles and Institutional Framework of Plan Nacer NATIONAL PROVINCE PROVIDERS GOVERNMENT ROLE/ Provides services Design Assessment RESPONSIBILITY and billing MANAGEMENT AGREEMENT Uses funds for: Per capita payment:  Single UMBRELLA AGREEMENT  Human  60 percent for Account Resources FINANCING enrollment Administration  Equipment  40 percent for  Service  Infrastructure health results payment  Supplies  Identification and Final validation of Registration Identification and ENROLLMENT National Registry  Administration Registration of provincial registry  Beneficiary registry Provider´s Records of medical VERIFICATION*  Tracers Performance record  Province´s Performance Source: Friedman, V. I., 2008. Note: Performance verification is different in the case of National and Provincial Governments. The National Government oversees Provinces in aspects such as financial performance, use of funds, incentive systems, counterpart funds, etc. Provincial Governments oversee providers in aspects such as reporting of information relevant to measuring service coverage (tracers), allocation of resources, compliance with protocols, invoicing, etc. The Umbrella Agreement establishes a monthly per capita payment in Argentinean pesos (ARS) for each registered beneficiary. The amount was originally ARS 12 and reached a final value of ARS 17 (approximately US$ 3). Of this amount, 60 percent is paid monthly based on the number of beneficiaries enrolled in the program. The remaining 40 percent is paid every four months and is based on the achievement of targets related to ten health indicators called “Tracers”. These targets are agreed upon by the Nation and the Provinces and are included in an annual performance agreement signed by both parties. The tracers enable the monitoring of the project’s progress and the development of the provincial insurance plans. These tracers have not changed since Plan Nacer started. They are described in Table 2. 12 Table 2 Tracer Definitions Health issue Description of the Tracer Operational definition of the Tracer rewarded measurement (every 4 months) I Early detection of Percentage of eligible pregnant Number of eligible pregnant women who received pregnant women women who received their first prenatal care during their first semester before 20 prenatal visit before 20 weeks weeks of gestation / Number of newborn infants to gestation. mothers eligible II Effectiveness of Percentage of Newborns (RN) of Number of Newborns (RN) of eligible mothers, in childbirth and eligible mothers with 5 minute the four month period, with five minute Apgar> 6 / neonatal care Apgar>6 Number of babies born to eligible mothers in the four month period III Effectiveness of Percentage of newborns (RN) of Number of Newborns (RN) of eligible mothers, in prenatal care and eligible mothers with birth weight the quarter, with birth weight≥2500g / Number of prevention of greater than 2500g newborns of eligible mothers, in the four month prematurity period. IV Effectiveness of Percentage of births to eligible Number of births to eligible mothers in the four prenatal and delivery mothers with Venereal Disease month period where the mother had care. Research Laboratory test (VDRL) VDRL testing in pregnancy and pre-natal tetanus / in pregnancy and pre-natal tetanus Number of newborns of eligible mothers, in the four month period V Case assessments in Percentage of case assessments Number of case assessments performed for child and maternal performed for maternal and child maternal and child (under one year of age) deaths deaths out of all child (under 1 year of age) deaths of total between the date of death and the last day of the and maternal deaths deaths. four month period / Number of maternal and child (under one year of age) deaths for mothers and children eligible that occurred in the four month period VI Immunization Percentage of eligible children Number of eligible children who turned 18 coverage under 18 months with measles months during the quarter that have received vaccine or Measles, Mumps and measles vaccine or MMR / Number of eligible Rubella Vaccine (MMR) children who are 18 months during the four month period VII Sexual and Percentage of eligible postpartum Number of eligible postpartum reproductive care women who received a sexual and women in the four month period who received at reproductive health consultation least one consultation for sexual and reproductive within 45 days postpartum health within 45 days postpartum / Number of newborns of eligible mothers in the four month period VIII Tracking of healthy Percentage of eligible children Number of eligible children less than 12 months on child up to one year under one year with complete the last day that was evaluated for the four month check-up schedule and percentiles period with complete control schedule, weight, size of weight, height and head and cephalic perimeter percentiles. from the start of circumference four month period in question / Total eligible children under 12 months IX Tracking of healthy Percentage of eligible children Number of eligible children between 12 and 72 child between 1 and 6 between one and six years months on the last day evaluated in the four month years with complete check-up schedule period with complete control schedule, weight and and weight and height percentiles size percentiles / Total eligible children 12 months or more and less than 72 months X Inclusion of the Percentage of providers serving the Number of providers that provide services to indigenous indigenous population with eligible eligible indigenous population, with at least one population personnel trained in specific care of trained on specific care of that population / Total that population number of participating providers in the Province Source: Plan Nacer Operations Manual 13 The second transfer of 40 percent of the per capita amount is paid every four months, based on achievements of tracer targets agreed upon explicitly between the National Government and each Province individually. It has gone through two iterations over time to measure tracer target achievement. At the beginning of the Plan, from 2004 to 2008, the 40 percent transfer was based on a binary achievement of that agreed upon level for each tracer. The percentage level was either achieved, in which case the 4 percent (that is, the total 40 percent divided by ten indicators) allocated to that tracer was paid, or it was not, in which case the payment was 0 percent. This was therefore an all-or-nothing scheme, whereby the assessment of the target achievement had only two possible outcomes: objective completed or not completed (see figure 1). Figure 1 Measurement of Tracer Target Achievement from 2004-2008 X axis: CAij/ Dij Where: CAij = Positive cases for tracer i in the province j Dij = Denominator for tracer i in the province j Y axis: CTij = % monthly payment for tracer i in the province j Source: Argentina National Ministry of Health, Plan Nacer. (2008). “Cambio en el modelo de gestión de trazadoras.” Serie de documentos técnicos nº 31. From 2008 onward, the all-or nothing scheme was replaced with a flexible, linear and continuous scheme. Each tracer now has three target thresholds: a minimum, an intermediate and a maximum value. As the Province crosses each of the three performance thresholds, that is, as the Province increases their target achievement, their payment is exponentially higher, because the slope of the curve determining payments gets steeper after each threshold. Therefore, for each of the ten tracers, achievement below the minimum threshold pays nothing, achievement between the minimum and intermediate thresholds pays between 2 and 3 percent of the per capita amount, achievement between the intermediate and maximum thresholds pays between 3 and 4 percent, and achievement above the maximum threshold (which is never below 90 percent coverage) pays 4 percent of the per capita amount (see figure 2). Therefore, Provinces are incentivized to produce the best health outcomes on each tracer to receive an exponentially high payment percentage for that tracer. For the ten tracers the maximum achievable remains 40 percent of the capitation payment. 14 Figure 2 Measurement of Tracer Target Achievement After 2008 X axis: SAij= Sanitary achievement for tracer i in the province Y axis: CTij = % monthly payment for tracer i in the province j Source: Argentina National Ministry of Health, Plan Nacer. (2008). “Cambio en el modelo de gestión de trazadoras.” Serie de documentos técnicos nº 31. In both cases of all-or-nothing or continuous exponential payment, all the tracer targets have equal weight. If at least four of the targets are not met in three consecutive periods, the national government has the right to declare the Province in violation of the Umbrella Agreement and suspend the transfers. In practice this right was used only in one case in 2007. Due to the two types and schedules of payments for the Results-Based Financing (RBF) program, linked to beneficiaries enrollment on the one hand and to tracers target achievement on the other hand, two verification processes are generated, one monthly and the other every four months. The schedule is the same for all the Provinces: January to April, May to August and September to December. It is also synchronized with the reports from the third party verification agent, in Argentina called the Concurrent External Auditor (CEA), hereafter referred to as the third party verifier. The money received by the Province per capita must be used exclusively to purchase services from providers. Under the Province-Provider Management Agreement, payment is made based on the provision of services registered and billed to the Provincial Insurance. Services are billed and paid according to a package of services which is the same in all participating Provinces. However, the prices of those services differ among Provinces given that they are determined by each Province. The provider is directly involved in making decisions about the end use of funds. The package of services was chosen for its impact on the health of the target population. In 2008 it included consultations and outpatient treatment, perinatal care, medical imaging, lab work, ambulance services, medical rounds in the community ("health rounds") and outreach services. In 2010, surgical correction of congenital heart disease was added to Plan Nacer’s benefits. In May 2012, health services for high-complexity neonatal care and high-risk pregnancy care were introduced. Later in August 2012, when Plan Nacer evolved into Programa SUMAR, the target population increased to include children up to nine years of age, adolescents up to 19 years of age and women between 20 and 64 years of age without explicit health coverage. 15 Finally, it should be noted that the Plan requires a joint financing effort, as each Province has been required to contribute a percentage of the per capita value since January 2009 in phase one and since January 2011 in phase two Provinces. This co-financing is verified periodically and penalties exist if the Province does not contribute accordingly. Penalties include an interest charge and eventually the suspension of transfers until Provinces comply with co-financing. In practice, every month half of the Provinces have some kind of financial delay. These delays are due to both financial and bureaucratic issues, but once the Provinces meet their contribution of funds, penalties are discontinued. INTERNAL ORGANIZATION OF PLAN NACER LINKED TO SUPERVISION AND VERIFICATION This section describes Plan Nacer’s regulation, organization, and information system, as well as the role of the third party verifier. In particular, it discusses issues related to the Regulatory Framework, the structure of the entities driving the Plan (organization, procedures, systems), and the system for supervision and oversight. REGULATIONS GOVERNING PLAN NACER The regulations governing the relationship between the National Government and the Provinces originate in the Loan Agreement, the Operational Manual, the Umbrella Agreements between the National Government and each of the Provinces, and the internal rules of the Unidad Ejecutora Central, the Central Executive Unit (UEC), which is now called the Equipo Nacional de Compras de Servicios de Salud, National Team for Purchasing of Health Services (UEC). The UEC has continuously made efforts to harmonize regulations, protocols and evaluation criteria, both in their definition and in the way they are supervised. These efforts were made to allow for a common interpretation of the rules and protocols across Provinces, as well as across different firms conducting the verification and audit of the scheme. Among regulations ruling program oversight, some trigger financial payments for their application or financial sanctions for their wrongful or lack of application; conversely, some oversight regulations do not have financial implications. Within regulations that have financial implications, there is a distinction between those regulations directly related to the measurement of results and those not directly related. Each type of regulation governs operational processes which have specific procedures and corrective and punitive consequences. Table 3 below presents the main aspects related to the operation and oversight of the Plan. 16 Table 3 Classification of Regulations Implemented by the UEC Type of Regulated operational Regulated aspects Corrective and punitive regulation processes aspects Regulation with Monthly capitation Definition of target population Sanctions for errors in the financial payments (enrollment allowable for registration enrollment registry at the implications results) provincial level: linked directly Enrollment tabulation process to beneficiaries with explicit to results fulfill certain requirements6 health coverage, duplication, incomplete data, inconsistent data, covered population. It Payment request process at provincial includes adjustments level and payment process (debit or credit) and fines Complementary transfers Determining dataset required to Sanctions for errors in the (subject to health results) justify each record of Tracer registry at the provincial completion (i.e. date of service, level: duplication, beneficiary identification, service data incomplete data, – that is, weight, size, etc.) inconsistent data, non- Acceptable data sources for field beneficiary population. It verification (for example, Medical includes adjustments record, Perinatal Information System (debit or credit) and fines (SIP)7, Vaccination record) Clinical record attributes (for example, doctor’s signature) Regulation with Processes linked to the Funds used exclusively to purchase It includes adjustments financial payment for services done services covered by the program at and fines implications not by the Unidad de Gestión the agreed prices related to results del Seguro Provincial, Payments made to providers with a the Provincial Insurance management contract in effect Management Unit Payments made to health facilities (UGSP) to providers used appropriately Payments made to providers within a specified timeframe Counterpart contribution Contributions made within a specified It includes fines and provided by the Province timeframe eventual suspension of transfers Actions with no Substantive fulfillment of Maintenance of the UGSP structure Regular incompletion can financial the agreed framework by the Province result in the suspension of implications Hiring of the providers by the the transfers and Province eventually in the Use of the funds by the providers in termination of the accordance with the Plan’s guidelines agreement Minimum achievement of the Tracers Source: Authors based on documents of the program, in particular the Operations Manual 6 Under the SUMAR Program, payments to Provinces require the provision of health services to a beneficiary in addition to enrollment. This is described under the “Effective Coverage” term. Effective coverage means that beneficiaries received a health service that was billed to and authorized in a given period of time according to each defined age group. 7 The Sistema de Información Perinatal, the Perinatal Information System (SIP) is an information system that fulfills international standards. 17 ORGANIZATION OF PLAN NACER For the implementation of Plan Nacer, the creation of specific organizations at the National Government level (initially the UEC, which was later called UEC) and in each of the Provinces was required. Annex 1 presents the organization of the initial UEC during the first phase of Plan Nacer. Beyond some minor changes that occurred over time, the UEC, naturally raising the number of people employed8, ran the project from the national level between 2004 and 2012. The responsibilities of the Coordination, Legal, Monitoring, Capitations and Operations Departments within the UEC are presented in Table 4. Table 4 Responsibilities in the UEC Departments Department Main responsibilities Coordination Relationship with the Provinces Overall management of the Plan Legal Contract enforcement Opinion on compliance with the contract Monitoring Overall supervision of processes with and without fiduciary impact Operational counterpart of the third party verifier General Verification of compliance with the overall goals of the loan supervision Verification of compliance with project agreements and regulations Verification of compliance with annual commitments Determination of penalties and incentives Satisfaction surveys of enrolled eligible beneficiaries Financial Audit of registers of eligible beneficiaries enrolled supervision Financial and accounting audit on the accounts of the SMIP Verification of compliance with financial content of the project Service Verification of field registries of beneficiaries supervision Verification of compliance with tracers Verification of services invoiced Verification of quality standards Capitation Transfer process Operations Corporate systems responsible for information processing Source: Authors based on documents of the program, in particular the Operations Manual INFORMATION SYSTEM The information system for the Plan has two basic sources of information: (i) one derived from the enrollment processes of potential beneficiaries; (ii) the other based on clinical records. Ideally, the best way would have been to define a single information system with standard functionality criteria that would have allowed interoperability across Provinces. However, the development of an information system followed a pragmatic path based on the idea that each Province should find the best way to generate the data, as long as the information provided to the Plan was totally standardized. The UEC designed Plan Nacer, and later Programa SUMAR’s information system and developed two analytical software packages: one to consolidate the beneficiary roster and one used to report tracers. The development of the two software packages and their operation are monitored and evaluated by the third party verifier, according to international standards. In its report for November-December 2011 based on surveys conducted up to then, the third party verifier firm, 8Initially there were about 40-50 people employed increasing to around 120 people by the end of Plan Nacer. 18 Crowe Horwath, rated the information system a two out of five in its level of maturity. However, there has been recent progress in several aspects and its maturity now rates a three. ROLES OF ACTORS INVOLVED IN VERIFICATION Together the UEC, in particular the Capitation, Operations and Supervision Departments, and the third party verifier are responsible for the verification process at central level. The roles of the different actors are described in Table 5. Table 5 Main Actors' Responsibilities in the Verification Process Actor Location Responsibility ex-ante Responsibility ex-post (before payment) (after payment) Verification of general Office- Capitation consistency of the data based and submitted by Provinces Operation Functioning of the Departments Field-based information systems at provincial level Analysis of the reports of the third party verifier Office- Analysis of the Supervision based scoreboard based on Department key indicators Verification of financial Verification of clinical records Field-based management Verification of the Counter-verification of general Office- functioning of central consistency of the data submitted based systems by Provinces Verification of enrollment forms, Third party Verification of service provision, verifier Verification of financial Field-based management, Verification of adherence to norms and procedures Source: Authors based on Operations Manual, interviews with UEC´s authorities and Terms of reference of the CEA MAJOR CHARACTERISTICS OF THE VERIFICATION METHOD WHO DOES THE VERIFICATION? The third party verification function is contracted through an independent firm 9. This contract is awarded based on international competitive bidding. The method of selection is based at 70 percent on quality and 30 percent on price. Contracts are signed for two years with an option to extend for another year if the government is satisfied. Once this is done, it is not possible to contract the same firm for another term for the same phase, so as to maintain the independence of the verification firm. There is a general rule, however, that the same firm could be contracted immediately for a different phase of the project, or that it could be contracted for the same phase 9Inaccordance with the Loan Agreement, it is not possible to disburse funds of the component related to capitation if there is no signed contract with a firm that performs the tasks defined for the CEA. 19 following a period during which a different firm is contracted. Overall, this rule has both positive and negative aspects. On the positive side, it results in a disruption in the relationship between the verifier and the one verified which reduces the possibility of familiarity and the risk of conflict of interest or collusion. On the other hand, there is a learning curve of at least six months for new firms, during which service quality might be affected and additional effort is required from the counterpart team. However, since the work does not start immediately upon the signing of the contract, the terms of reference (ToR) typically include initial tasks of retroactive review for operations carried out in the brief periods before contract signing, so the firm can get more familiar with its expected duties. The contract is a fixed price contract. Ten percent of the price is paid in advance, and the remaining amount is paid through bimonthly installments, upon the issuing of the reports. THE SCOPE OF WORK OF THE THIRD PARTY VERIFIER The third party verifier was envisioned as a tool to audit the main fiduciary aspects of the Plan, create a culture of verification between various stakeholders and ensure an unbiased and independent opinion in the case of disputes, mainly between the State and the Provinces. The third party verifier is primarily responsible for the verification of information submitted by Provinces to the National Government, and for the review of information processes conducted in the UEC with this information. Specifically, the third party verifier firm must counter-verify beneficiary enrollment data (following verification by the UEC), verify that the tracer data was valid, ensure compliance by the Provinces with general procedures, and report whether payments to providers complied with standards. During phase one and two, different firms were hired. The ToR were adjusted over time, either to improve the precision of the requirements or to adapt to changes in the Plan, however between 2005 and 2008 the ToR were not substantially modified. The ToRs are summarized in Table 6. Table 6 Third Party Verifier's TOR Details General To verify that the data on the eligible population and service delivery is a reliable basis objective for the transfer of national resources to the Province, and that the data is therefore also a valid basis for the financial audit. Specific  Issue a professional opinion on the integrity, validity, and consistency with the objectives objectives of Plan Nacer of the procedures used by each Province for the settlement of and payment for services.  Issue a professional opinion on the integrity, validity and consistency with the objectives of the Program, for billings justifying expenditure presented by the Provinces to the UEC of the National Ministry of Health, including billing based on beneficiaries enrolled (60 percent of total payment) and the tracers-based billing (40 percent).  Issue a professional opinion on whether the payments made by the UGSP to the providers has complied with the rules of the program, for example, on whether the payments are made only to contracted providers, whether they correspond solely to the services contained in the package of services of Plan Nacer, whether such services were actually provided, and whether the services are billed at the price defined in the package of services.  Issue a professional opinion on the completion of conditions set for the program in the Loan Agreement and in the agreements signed between the Provinces and the State, as well as the contracts between UGPS and the providers. Scope  Verify that the respective Province is submitting to the UEC the registry of Obras Sociales Provinciales, Provincial Social Works (OSP) and that it is acceptable to the UEC.  Verify data consistency in the SMIP list of enrollments (“registry”) for each 20 Details participating Province by cross-checking the registry of the Obras Sociales Nacionales, National Social Works (OSN) with the registry of the OSP.  Verify the existence of contracts between providers and UGSP, and that they conform to the guidelines of the program.  Verify that the payments from UGSP to the providers are made based on services provided, the prices of which are based on production factors (wages, human resources contracts, investments in equipment or infrastructure, or other drugs).  Verify the accuracy of the billings submitted by the Provinces to the UEC justifying the monthly transfer payment (60 percent of the per capita payment).  Verify the accuracy of the billings submitted by the Provinces to the UEC on the achievement of the tracers (up to 40 percent of the per capita payment).  Check the accuracy, timeliness, and compliance of the transfers from the UEC to the Provinces, in line with the terms of the program.  Maintain close liaison with the financial audit, done by the Auditoría General de la Nación, National General Auditor (AGN).  Issue periodically (at least every four months) conclusions and recommendations for improvement of the processes involved in all of the above scope items. Source: Authors, based on the ToR of the CEA Since 2009 there have been some minor changes in the ToR. First, there are certain changes in the objectives and formal aspects to further specify responsibilities and describe all activities. For example, the fourth objective that was referring to a vague general task was replaced by two more concrete objectives as follows:  To issue a professional opinion on whether participating providers were contracted by UGSP in accordance with the contracting rules defined by the program.  To issue a professional opinion on whether the application by the UEC of any of the sanctions provided by the program for participating Provinces was in accordance with the rules of the program. Also, it was explicitly stated that it was mandatory to issue both an Opinion and an Internal Verification Report. The Opinion Report is presented in a short report used for decision making; it describes verification results, fiduciary information and proposes penalties to apply to Provinces. The longer, Internal Verification Report describes whether the different actors fulfill their obligations as set out in the program, and provides a review of organizational aspects, processes, and compliance with formal project documents. This separation of the two reports has been done in accordance with an agreement between the firms and the authorities. Some responsibilities were also added related to the extension of the Plan to cover the surgery for Congenital Heart Disease. It is important to mention that the ToR of the third party verifier is primarily focused on the issuance of a third party, independent opinion. This results in a peculiar relationship with the authorities of the Plan and it should be noted that:  In order to provide an independent view, the firm must have the autonomy to select and apply the most appropriate methodology to support its opinion.  As a general rule, the opinions issued by the firm are considered binding by the authorities of the Plan, although there are a few exceptions such as explicit error from the firm representative on a sanction to a Province, or the submission by the Province of a report justifying that the sanction requested by the firm is not appropriate. The third party verifier’s report is written every two months (results of tracer verification are reported in every other report). There is regular discussion of the third party verifier´s reports. Eventually Provinces involved can participate in the clarification of the points established by the third party verifier, by submitting requests for clarification as per the program’s operational manual. The operational manual also states that the UEC can evaluate the opinions issued by the firm and act as the firm’s technical counterpart. 21  Due to the professional relationship between the third party verifier and the authorities of the Plan, the approvals or rejections of the reports are based on highly objective criteria, and do not affect the independence of the firm. HOW IS VERIFICATION DONE? The firm Crowe Horwath was first hired for phase one. It was subsequently contracted for phase two and for the audit of the Congenital Heart Disease Program. During the execution of the first contract, UEC and Crowe Howarth developed tools and reports to carry out verification and counter-verification, as well as financial audit tasks. These tools have been included in the ToR for subsequent tenders and used by the firms contracted in later contracts 10. The above- mentioned Opinion and Internal Verification Reports stemmed from this tool development by the first firm. In the next sub-sections, the following processes will be reviewed individually:  Beneficiary Enrollment Verification  Beneficiary Enrollment Counter-Verification  Tracer Verification  Verification of the Use of Provincial Funds to Purchase Services  Other General Verification Verification of Beneficiary Enrollment The first level of verification of beneficiary enrollment is the responsibility of the Capitation Department of the UEC and involves the following procedures:  Every month, the Provinces (UGSP) send a CD to the UEC with the full archive of data on the provincial registry (File B). This information supports the capitation payment request made by the Province to the National Government.  The UEC consolidates the files sent by all the Provinces to purge duplicate records (there is a possibility that a beneficiary appears registered in more than one Province, in which case there are preset criteria to detect the valid record) and produce a unified national registry of the Plan.  This registry is compared against the largest dataset of the explicitly insured population in the country, which comprises the beneficiaries of the National and Provincial Social Security Insurances and other programs with national coverage, and is called Padrón Único Consolidado Operativo, the Unified Consolidated Operating Registry (PUCO). This comparison is done to verify that registered beneficiaries are not already insured and are eligible for the Plan.  The results of the verification carried out by the Capitation Department are translated into adjustments to the payment amount, and when applicable, corresponding fines are applied. Not all adjustments result in fines; for example, when a beneficiary is registered in more than one Province, because one Province does not know what happens in another Province, neither will be fined. When finalized, the adjusted capitation payment transfer is made. The Operational Manual for Plan Nacer details the fines applicable under various scenarios. 10In all cases, firms hired used as general methodological approach the recommendations of COSO (Committee of Sponsoring Organizations of the Treadway Commission - www.coso.org) 22 Counter-Verification of Beneficiary Enrollment The second level of verification is done by the third party verifier and follows a set of procedures in which there is a distinction between tasks performed in the Provinces and those performed in the central offices. The results are given to the UEC in a report every two months. Tracer Verification Tracers measure performance against select maternal and child health indicators, which determine part of the capitation payment (up to 40 percent) to the Province. The numerator uses data reported by the facilities11 to the province, where data is collated and reported to the national level while the denominator at provincial level is based on official information which is updated every 1-2 years. In making its determination, the third party verifier uses confidence intervals to analyze the results. The verification of achievement against tracers is performed every four months and, as in the case of the List of Beneficiaries, the process begins in the Capitation Department of the UEC. The Capitation Department receives an Affidavit from the Province reporting the level of achievement, with supporting data on CD. Afterwards the third party verifier initiates the process of verification (see figure 3). It consists of three steps: (i) general consistency test, (ii) information source validation, and (iii) field test. The first two are done on every case presented by the Province, whereas the last one is done on a sample of cases. 11 As the facilities are paid on a fee for service basis by the provinces and not based on targets. 23 Figure 3 Third Party Verifier's Tracer Verification Process Reception of Affidavit Set of tracers declared by Provinces as fulfilled Inconsistent Cases Global Consistency Proof Set of tracers declared fulfilled with consistent data Positive cases Source´s origin Proof of origin of the presented by source ineligible providers Set of tracers declared fulfilled with consistent data from appropriate Providers Cases with wrong or incomplete data Proof of Verification in the field Proof Set of tracers declared fulfilled and verified by CEA Source: Authors, based on Crowe Horwath Internal Verification Bimonthly Report and SMIP The general consistency test involves a series of checks on the database of positive cases sent by the Province, in order to detect the following: duplication, inconsistent data, missing data, verification against the PUCO, and consistency of the dates. Details of the methodology used for global consistency can be found in the bimonthly report submitted to the UEC. This methodology provides information regarding to whom it applies, criteria for detecting duplicates, minimum data requirements for registries, and allowable consistency parameters for data. Subsequently the third party verifier performs information source validation to ensure reported data originates from approved sources. Possible information sources include the UGSP, a hospital, or Centros de Atención Primaria de la Salud, Primary Health Care Centers (CAPS)12. Finally, there is field verification. The previously described processes determine which of the cases presented by the Provinces are valid. Therefore only the valid data from the Tracer records 12While data for tracer I, VI, VIII and IX (see table 2) comes from CAPSs, data for tracers II, III, IV and VII comes from hospital databases. Data for tracers V and X comes from the UGSP. 24 that the Province declares completed constitutes a pool from which a field verification sample is drawn. The issue of sampling will be discussed further, but at this point it should be noted that there are standards for quality and criteria that the data must meet (the minimum criteria is to have a registry, acceptable sources for comparing data, registry characteristics). The field test carried out by the third party verifier consists of a review of clinical records and data quality at each sampled health facility following guidelines and protocols defined by the Ministry of Health. The existence of clinical records with all required information filled out, and the consistency of the information, allows one to estimate, to a degree, the quality of care, which is not audited directly. The results of the tracer verification activities are reported to the UEC in every other Opinion Report, every four months. Verification of the Use of Provincial Funds to Purchase Services The third type of verification done by the third party verifier has to do with the use of the funds received by the Province through the monthly capitation (60 percent) and additional tracer payments (up to 40 percent). The verification of beneficiary enrollment and of tracer indicator achievement described so far have to do with the fiduciary oversight of the funds transferred by the UEC to the UGSP. Here we are describing the fiduciary oversight of the funds transferred by the UGSP to the providers. Strictly speaking, the verification of the payment for results could be considered finished after verification of the beneficiary enrollment registries and Tracers, since the RBF contract is between the National level and the Province. However, one specific rule of Plan Nacer is that the funds must be used to purchase services covered by the program actually provided to beneficiaries. The UGSP is primarily responsible for overseeing that this is the case. Their oversight is verified by the third party verifier, because the National Government wants to ensure that the funds are transferred in accordance with the rules of the program. For this level of verification, a sample of the paid services is taken and verified in the field, in the same way as it is done with the records that are used to track the achievement of the Tracers. Verification samples tend to have a significant overlap with the samples used to check the achievement of Tracers due to the fact that: (i) providers that report records for Tracers usually also have the capacity to bill services, so the set of providers tend to be the same in both cases; (ii) there is an implicit incentive for the third party verifier to match both sets of providers because this means lower costs for verification. Other General Verification Operational Monitoring The above-mentioned verification mechanisms are described in the Opinion Report addressed to the UECENCSS, and have direct financial consequences for Provinces. The report is generated bi-monthly and presents the findings of the verification of registries and of the payment to providers. Every other report also includes results on the tracers. This report also includes a proposal of the penalties to be applied to the Province as result of the assessment carried out. In addition to these verifications, the third party verifier performs a set of general monitoring tasks that ensure good professional practices and are summarized in the Internal Verification Report. These tasks include ensuring compliance with agreements and administrative requirements, as well as establishing protocols (see Annexes 2 and 3 for details). 25 Evaluation of the Work of the Third Party Verifier and Reports Approval The Opinion Report and the Internal Verification Report are reviewed by the relevant departments of the UEC, which are: Capitation, Legal, Supervision, and Medical. Whenever there is a discrepancy related to compliance with the ToR, the clarification is issued and, if resolved, the report is approved to trigger the payment for services provided by the third party verifier. The final reports are resubmitted to the respective Provinces. At the same time, the reports are submitted to the AGN, which performs an annual ex-post financial audit. Up until now, the AGN has not had any negative comments regarding the financial functioning of the Program. 26 VERIFICATION RESULTS VERIFICATION RESULTS ON BENEFICIARY ENROLLMENT Table 7 shows the degree of rejection of the list of beneficiaries’ records from 2004 to 2012, calculated by the authors. It distinguishes between the Provinces of the two phases of the Plan, as reported by the third party verifier. The data presented are the yearly sum of the monthly records declared and rejected, considering that verification is done on a monthly basis. Table 7 Records Processed and Rejected- Plan Nacer 2004 2012 Item Phase Year Total 1 2 Records Processed (N) 42,391 42,391 2004* Records rejected (N) 8,622 8,622 Rejected/Processed (%) 20 20 Records Processed (N) 2,392,936 2,392,936 2005 Records rejected (N) 64,815 64,815 Rejected/Processed (%) 2.7 2.7 Records Processed (N) 4,706,802 4,706,802 2006 Records rejected (N) 4,934 4,934 Rejected/Processed (%) 0.1 0.1 Records Processed (N) 5,586,187 1,198,125 6,784,312 2007 Records rejected (N) 9,809 7,155 16,964 Rejected/Processed (%) 0.2 0.6 0.3 Records Processed (N) 6,331,941 4,059,562 10,391,503 2008 Records rejected (N) 7,637 24,893 32,530 Rejected/Processed (%) 0.1 0.6 0.3 Records Processed (N) 6,329,245 5,681,373 12,010,618 2009 Records rejected (N) 9,529 24,233 33,762 Rejected/Processed (%) 0.2 0.4 0.3 Records Processed (N) 6,603,221 9,446,180 16,049,401 2010 Records rejected (N) 25,260 58,641 83,901 Rejected/Processed (%) 0.4 0.6 0.5 Records Processed (N) 6,799,650 13,104,489 19,904,139 2011 Records rejected (N) 69,556 135,829 205,385 Rejected/Processed (%) 1 1 1 Records Processed (N) 4,072,260 8,758,807 12,831,067 2012** Records rejected (N) 21,917 60,527 82,444 Rejected/Processed (%) 0.5 0.7 0.6 Records Processed (N) 42,864,633 42,248,536 85,113,169 Total Records rejected (N) 222,079 311,278 533,357 Rejected/Processed (%) 0.5 0.7 0.6 * Only for December, month of starting up of Plan Nacer ** Period of January-July only Source: Authors, based on information from the Capitation Department of the UEC 27 As can be seen, in the first phase of the Plan, there was a sharp drop in the rate of rejection during the first two years. Over the years the rejection rate in phase 1 Provinces then stabilized at levels in the range of 0.4-0.6 percent. In phase two, the rejection level was slightly more stable than in the first phase. The rate of rejection is evidently very small and is reflected in the very small level of financial adjustments made to beneficiary enrollment payments. It should be noted however that these adjustments are based on database verification and not field verification. VERIFICATION RESULTS ON TRACERS The verification of tracer indicators determines the degree of achievement for the tracers. This degree of achievement is a percentage applied on the 40 percent of capitation payment to determine transfers. The third party verifier assesses the achievements of the Provinces for each of the ten indicators against their declared figures. In the instance that the third party verifier deems certain patient records not to meet the required criteria13 for a particular indicator, those records are rejected. These rejected records will reduce proportionally the amount received by the Provinces. This adjustment of the amount received by Provinces has amounted to 11.5 percent of total tracer payments over eight years (see table 10). For the purpose of this study, the authors processed and standardized information provided by the Plan in order to analyze the first and second phases together. As a result, for each four month period, the tracer achievement percentage declared by Provinces and the percentage determined through the third party verifier assessment were calculated. Figure 4 shows the percentage of declared records (that is, records which the Provinces state meet the criteria for a tracer) which are then rejected by the third party verifier. Ideally there would be no difference between the declarations made by the Provinces and the assessment of the third party verifier. The graph below shows the rejection rate started at 25 percent in 2005, and slowly decreased, albeit not in a smooth progression, towards 5 percent. Figure 4 Percentage of Declared Records Rejected by the Third Party Verifier Source: Authors 13 The criteria are described in the Operations Manual of the Program 28 The following two tables 8 and 9 are based on data from all Provinces from the second trimester of 2008 up to the second trimester of 2012. Taking all the tracers for all Provinces and all trimesters between 2008 and 2012 together as the universe of results, table 8 shows that of the total number of tracers declared, a significant proportion showed a greater than 20 percent difference between declared and verified, whereas a relatively small proportion showed a greater than 40 percent difference between declared and verified. This shows that whilst declared results are sometimes erroneous, the majority of declared results are relatively accurate. The value of third party verification is though proven to be valuable. As can be seen from table 8 as well, the incidence of error rates varies by indicator. Table 8 Error Rates Higher than 20 and 40 Percent by Indicator, 2008-2012 Tracer Percentage of results Percentage of results with error rates >20% with error rates >40% of declared (%) of declared (%) I Early detection of pregnant women 23 II Effectiveness of childbirth and neonatal care 6 1 III Effectiveness of prenatal care and prevention of 7 1 prematurity IV Effectiveness of prenatal and delivery care 19 5 V Case assessments in child and maternal deaths out of all 5 3 child and maternal deaths VI Immunization coverage 10 1 VII Sexual and reproductive care 18 6 VIII Tracking healthy child up to one year 11 3 IX Tracking healthy children between one and six years 24 7 X Inclusion of the indigenous population 6 2 Source: Authors Note: Some tracers have incurred higher error rates due to tracer-specific challenges. For example, tracer IX has incurred high error rates due to the absence or incompleteness of clinical record and registers. Efforts have been made to reduce error rates over time, including through improving data recording. Table 9 shows how often the achievement of a Province for one tracer was downgraded by one tier following verification by the third party verifier. That is to say, for example, that a Province, following third party verification, would have been downgraded from the tier which pays between 3 and 4 percent of the capitation payment, down to the tier which pays between 2 and 3 percent of the capitation payment. Again the table shows that there is significant variation across tracers. Table 9 Percentage of Cases in Which One Province Moved Down a Tier for One Tracer Following Verification Tracer Percentage of verified results moving achievement down a tier (%) I Early detection of pregnant women 14 II Effectiveness of childbirth and neonatal care 5 III Effectiveness of prenatal care and prevention of prematurity 6 IV Effectiveness of prenatal and delivery care 20 V Case assessments in child and maternal deaths out of all child and 3 maternal deaths VI Immunization coverage 7 VII Sexual and reproductive care 8 VIII Tracking healthy child up to one year 7 IX Tracking healthy children between one and six years 23 29 Tracer Percentage of verified results moving achievement down a tier (%) X Inclusion of the indigenous population 7 Source: Authors Note: This table calculates the difference between the payment for each indicator based on declared coverage, and the payment based on verified coverage. Where the difference in payment is 1 percent or greater (out of a possible 4 percent per indicator), it is assumed the achievement dropped a tier . The verification process involves many actors and each of them can make adjustments both related to beneficiary enrollment and to tracer indicator achievement. For example, after adjustments in payments in the Capitation Department over the official declaration of a Province, there can be further adjustments made by other departments of the UEC or by the third party verifier, for achievements that originated in the previous periods (because the third party verifier verifies post-payment). The adjustments are communicated to the Provinces, which may appeal and settle the matter with the authorities of the UEC. Most of the adjustments assessed in the field pertain to tracers. In this respect it is important to remember that the determination of errors done by the third party verifier is based on a sample. However, the error percentage determined on this sample is then used to determine the Province’s achievement rate for that indicator. This estimation takes into account a confidence interval calculated based on the parameters with which the corresponding sample was designed. ADJUSTMENTS AND FINES BASED ON VERIFICATION The Operational Manual of the program includes a set of situations that, if identified, trigger adjustments and penalties to the capitation payments made to the Provinces. In case of serious breaches, the contract with a Province can be suspended or terminated. Suspensions of transfers have been rare (and all related to non-contribution of Provinces to the joint funding of the Plan, rather than for RBF-related reasons) and no contracts have been terminated. The usual processes of supervision and verification by the UEC and the third party verifier have led to the application of adjustments and fines enumerated in the Operational Manual, as well as actions that may arise from verification findings, both from internal parties and the third party verifier. Table 10 presents a summary of the total adjustments over the project. 30 Table 10 Adjustments and Fines Related to Payment Transfers from 2005-2012 Percentage Department of Internal of Third Party Totals enrollment audit Others Concept Capitation Verifier and tracer (Supervision) Payments transfers (%) United States Dollars A: Gross monthly transfers (beneficiary enrollment) 189,142,370 Adjustments by systems (64,172) (201,594) (648,025) (913,791) 0.48 Adjustments in the field (134,842) (1,697,985) (1,832,826) 0.97 Other adjustments (4,986) (4,986) Adjustments subtotal (64,172) (336,436) (2,350,996) (2,751,603) 1.45 Fines (2,748,915) 1.45 B: Net monthly transfers (beneficiary enrollment) 183,641,852 C: Gross complementary transfers (tracers) 72,266,020 Adjustments 97,853 175,279 (8,278,030) (304,557) (8,309,454) 11.5 Fines (1,595,092) 2.2 D: Net complementary transfers (tracers) 62,361,473 Total gross transfers (A+C) 261,408,390 Total net transfers (B+D) 246,003,325 Total adjustments and fines (15,405,065) -5,89 Source: Authors, based on data from Capitation Department of the UEC Note: Amounts are translated to US$ from ARS at an average rate between April 2005 and August 2012 of 0.2885 US$ to 1 ARS. Amounts in parentheses are negative. The above table reflects a number of interesting features:  Fines and adjustments represent 5.9 percent of gross transfers recorded from 2004 to June 2012.  The majority of adjustments are related to tracer payments accounting for US$8.3 million versus US$2.75 million for beneficiary enrollment adjustments.  The majority of the adjustments followed third party verifier assessments and, within them, most are based on findings from the field. SAMPLING The work of the third party verifier is centered around issuing an opinion on main fiduciary risks of the Plan. In this context, the methodology developed by the third party verifier ensures oversight of the major financial bodies of the project. Once the information systems had been sufficiently tested, the main sources of audit findings were the third party verifier field work. The process of verification for the third party verifier involves three types of samples: 31  Samples of enrollment forms: These are the samples that are drawn bimonthly from the beneficiary roster. The third party verifier verifies the consistency of the information and the compliance with certain formalities such as the beneficiary signature in the application form.  Samples of tracer achievement information: A representative sample is taken for each Province and for each tracer from the provincial level records to determine the degree of achievement against the target set for the tracer. Sample selection takes into account the geographical areas and health facilities that present and contribute to the highest volume of records (thereby constituting risk-based sampling).  Samples of services billed by providers to the UGSP: In each Province a sample of services billed by the providers is checked. From a financial point of view, the most important sample is the one linked to the completion of results, that is, the sample of tracer information. While the other two samples generate findings exclusively from cases encountered and are more related to ensuring administrative requirements and appropriate use of funds, samples of tracer information are used to estimate overall achievement rates. The following points relate to this last type of sample. The firms base their professional opinion on a sample with a size and distribution that makes their opinions technically sustainable, but that are also financially and logistically optimal for the third party verifier (the more facilities are seen, the more expensive and logistically challenging the verification exercise becomes). By contrast, from the standpoint of the authorities of the Plan, to ensure adequate oversight and accurate reporting, it is important to maintain on all health facilities a credible threat of verification, and hence of sanction or adjustment should the clinics misreport on their achievements. In other words, it is important for the authorities of the Plan that all health facilities be included in the sampling frame used by the third party verifier, even if not all facilities may end up in the sample drawn from that sampling frame. Determining the optimal sample implies an economic calculus that considers items such as the number of clinics to visit, the distance from the clinic to the CEA´s headquarters, or the number of records that each clinic reports. Until now, the fiduciary framework has let the third party verifier with a certain level of freedom in how the sample of health facilities is determined to ensure the independence of its opinion. However, the TOR does require a minimum of accuracy/representativeness. The methodological criteria used for deciding on the sample are the same in each province – however, the size of the sample is different in each case. Based on the experience of a third party verifier in 2011, samples highlighted the following: (i) the number of visits to clinics represented 15.1 percent of the total number of clinics that reported production; (ii) the number of different clinics visited represented 10 percent of the total number of clinics that reported production; and (iii) the estimated distance from the clinics involved in the samples to CEA´s headquarters represented 9.5 percent of the distance required to visit all the clinics that reported production. This data implies that it is possible some remote, small health facilities may not be included in the sampling frame. As a result, the fiduciary responsibility of those facilities may be diluted, since they may never be subject to the verification of tracer achievement. Improving the information system so that verification can be done by the third party verifier without physically reaching all facilities could be one way to increase the fiduciary responsibility of all facilities, at an affordable cost. In addition, this point highlights that the role and responsibility of the provinces, as opposed to the UEC, in ensuring accurate results by health facilities may still be a question for the program to explore further. DISSEMINATION PROCESS Once they have been internally validated by the UEC, the reports of the third party verifier and the UEC Supervision Department are sent to the respective Provinces for their acknowledgement, with an opportunity provided to discuss the results. The reports are also sent to the AGN that 32 performs an annual ex-post review of the main fiduciary aspects of the Plan, as well as its general evaluation. There is generally no additional dissemination of the information contained in the reports. COST OF THE THIRD PARTY VERIFIER To analyze the costs of the third party verifier, several elements could be considered that can serve as general parameters in this work. It should be considered that so far four firms have been contracted by the Plan to carry out the third party verification. These are Crowe Horwath, Price Waterhouse, PKF and Bértora. The first one has the most years of experience. It is also the only one that audited both phases of the program. The contractual values of the selected firms are described in Table 11. 33 Table 11 Contract in US$ for Different Phases and Portions of the Project Cost per Duration Amount in Cost per month Contract Start End Description Phase in US$ month in per status date date months without tax US$ province in US$ Third party verifier APL1 Closed Apr-05 Mar-07 24 1,651,700 68,821 7,647 Third party verifier APL1 Closed Jul-07 Jun-08 12 1,146,186 95,515 10,613 (extended I) Third party verifier APL2 Closed Jul-08 Dec-08 6 459,249 76,542 8,505 (extended II) Third party verifier APL1 Closed Jan-09 Apr-09 4 277,438 69,359 7,707 (extended III) External APL1 Closed Jun-09 Jun-11 20 2,300,000 115,000 12,778 Concurrent Audit External Concurrent Audit APL1 Closed Jul-10 Jul-11 12 110,000 124,167 13,796 APL 1 Congenital Heart Disease External Concurrent Audit APL1 Active Dec-11 Dec-13 24 4,693,476 195,562 21,729 NOA-NEA External Concurrent Audit APL2 Closed Sep-07 Sep-09 24 4,235,000 176,458 11,764 APL 2 External Concurrent Audit Congenital Heart APL2 Closed Jul-10 Sep-11 13 630,857 224,986 14,999 Disease APL 2 provinces External Concurrent Audit APL2 Closed Sep-09 Sep-11 24 4,220,962 175,873 11,725 Second Phase External Concurrent Audit APL2 Closed Sep-11 Sep-12 12 2,589,693 215,808 14,387 Second Phase (extended) Source: Authors, based on data from Legal Department of the UEC. The total cost is related to the length (in months) of the contract and the number of Provinces audited. An appropriate measuring unit is the cost per Province or per month of contract. One reason for that is that the verification tasks did not involve any notable differences in personnel assigned to the Provinces (perhaps with the exception of the Province of Buenos Aires which is significantly higher than the rest), considering that the sample sizes are relatively similar. Based on the average value of the contracts it can be concluded that the average cost per Province and per month is in the range of $12,332 dollars, and the amount is very similar for both phases of the Plan. 34 There is one additional variable that could influence the cost: the dispersion of field samples. In this respect there are significant differences across Provinces (for example, more dispersed populations in Southern Provinces). However, because samples are usually drawn from relatively dense Province areas, there does not appear to be a significant cost difference among Provinces. The last element to consider with regard to bidding is that the bidders face a two-year agreement, which in Argentina carries a macroeconomic risk. Consequently, the bids are expressed in US dollars, but during the analyzed period the exchange rate grew at a lower rate than that of the salaries. Assuming that bidders make some allowance for the expectations they have at the time of signing, the values are possibly somewhat distorted. The costs of the contracts are financed by the Project; there is a special budget line for this with the understanding that the verification process is a transactional cost inherent to the RBF approach. The composition of the verification teams could also affect the cost. A typical team consists of a socio-accountant, a medical manager, an accounting manager, medical professionals, accountants, assistants, and an information systems staff member, as presented in table 12. Table 12 Third Party Verification Team Composition Function Number of people Time allocation Profile Overall direction 3 Part-time 1 Socio Accountant 1 Medical Manager 1Accounting Manager Central supervision 1 for every 8 Provinces Full-time Medical Field supervision 1 for every 4 Provinces Full-time Medical Medical auditor 1 for each Province Part-time Medical Accounting-organizational 1 for each Province Part-time Accountant auditor Central support staff 1 for each Province Full-time Assistant Systems 1 for every 5 Provinces Part-time Information Systems Source: Authors, based on interviews with experts and Crowe Horwath staff Note: in the case of the Buenos Aires Province, there are 2 or 3 staffers for each category In quantitative terms, this structure requires approximately 310 consultancy hours per Province per month. However, the contract for Phase 2 which included Buenos Aires, required 350 hours. Freight and transportation costs, which depend on the dispersion of the samples, fall at an estimated 12-15 percent of the total contract cost.14 Plan Nacer´s management requires high quality information which means accepting higher transaction costs. Among these costs are those associated with the third party verifier. When comparing the total amount of the third party verifier contracts with the amount of the subcomponent corresponding to the RBF payment (consisting of the capitation amount of Component 1 of the project), it appears that this amount represents 6.69 percent of the total. 14 Information from meetings with consulting firms 35 Table 13 Cost of Third Party Verifier Capitation Subcomponent financing amounts (in million US$) 298.4 CEA contract amounts (in million US$) 19.96 Share of CEA contracts in Capitation subcomponent financing (%) 6.69 Source: Authors, based on Plan Nacer Project Appraisal Document for phases one and two and information from Legal Department of the UEC. The third party verifier tasks involve some indirect costs for the necessary involvement of UEC staff. It is highly difficult to estimate these costs and separate operational management tasks for specific departments of the Plan. However there is an allocation of eight accountants in the Supervision department, four people in the medical field, four people in the Capitation department, one in the Operations department, and finally the Legal department has three lawyers. The time allocation of this staff is variable. All are involved for one to two weeks every two months to review the reports of the third party verifier. Supervisory staff also has a continuous task of developing its management supervision plan, which implies devoting half of the time in the field and the rest in the offices of the Plan. Medical staff is strongly involved in field tasks, but dedicate their time to developing best practices to improve service quality. SUMMARY AND CONCLUSIONS Plan Nacer has been developing since 2004, with a strong focus on continuous improvement. It was never regarded as a static, finished product but as a constantly evolving project. This evolution has been fairly natural as different actors keep adapting to the changes that occur. This sense of evolution can be seen in small adjustments to the regulations, which were always intended to be few, clear and accomplishable. This resulted in various actors adapting their procedures and behaviors to these regulations. This organizational environment led to the development of a dynamic integration of operational and supervision/oversight roles and procedures. From the operational point of view, the Capitation and Operations department bear the biggest responsibility, as they are involved in receiving documentation from the Provinces that triggers the process of payment, verifying the information through the available means, making appropriate adjustments, and processing the respective payments. On the other hand, the supervision, oversight, verification and counter-verification processes are to be found primarily in the Supervision department and in the third party verifier, the Concurrent External Audit. A third party audit in a classical sense is the one made annually by the National General Auditor. There are some observations that could be considered key to the performance of the verification processes in the Plan:  The regulatory framework of the Plan displays strong internal consistency between regulation, structure, information system and the verification framework.  Verification processes started out with a certain degree of overlap, to later become more specialized in terms of supervisory tasks, which in turn helped the actors to better understand these processes. 36  The smooth interaction of operational and monitoring processes generated stability in the processes that facilitated the incorporation of the concurrent work of the third party verifier. There are several important points regarding Plan Nacer:  The Plan aims to create an environment that ensures the trust of all the stakeholders by having clear, simple and achievable rules.  Some overlap in the processes carried out by the Supervision department in charge of the program’s internal oversight and those carried out by the third party verifier were built in from the beginning. This overlap was useful to complete and standardize processes and procedures, for staff training and for better management of the contract with the verification agent. Once processes were clearly established, the overlap was reduced over time, generating a functional specialization, which leads to less redundancy and more efficient processes.  Likewise, these processes were designed with a focus on operational monitoring. This meant that while verification happens after the fact, it is done shortly after the results and processes have occurred, which allows more accurate and effective corrections to be made. In line with this focus, an auditor (the third party verifier) was contracted to fulfill their role concurrently.  After eight years of evolution, Plan Nacer has developed processes and procedures with a high level of sophistication and formalization, particularly those related to measuring results. Those processes are now documented and described in several procedural manuals. Also, payment processes are ISO 9001 certified. As an institution of the project, the third party verifier has played a very important role. It has contributed positively to Plan Nacer in a number of ways:  By helping create an environment for oversight of the Provinces;  By acting as a mediator to settle differences between the actors, namely between the National Government and the Provinces;  By contributing to the improvement of the standards of Plan Nacer, ensuring that reporting errors, both unintentional and intentional, are corrected;  By providing inputs to the management of the Plan, through the Coordination unit of the program;  By identifying 70 percent of adjustments and penalties, this has equaled about 37 million pesos from the beginning of the plan until June 2012. The third party verifier has two features that are relevant and somewhat contradictory for a classic auditor. First, it is "Concurrent" which puts it very close to the management and translates into engaging in some operational processes such as determining adjustments and penalties. Second, its independence as a third party forces it to distance itself from the other actors. The desired balance would be to remain close enough for management to develop a profound understanding and distant enough to be credible as an independent opinion in case of conflict. The above issue presents a challenge for firms that are hired by the Plan, as they are generally accustomed to a strict role of audit firms. It is also a challenge for the authorities of Plan Nacer, who must maintain a climate of control and ensure balance between involvement in management and an independent opinion. This balance can evolve over time, as the verification objectives themselves evolve along the spectrum of control or fiduciary responsibility on the one hand, and supervision and system strengthening support on the other hand. After an international competitive bidding process, the winning third party verification firms are contracted for two years, with the option of an extension for another year. The changes of the verification firms are often accompanied by periods of increased demand for additional UEC support as companies that are new to the Plan have to make significant efforts to assimilate the 37 internal logic of their processes to the audits. The benefits of change are linked to the possibility of an unwanted strengthening of the bond between the third party verifier and the Provinces. However, this benefit should be weighed against the costs generated by the changes. The ToR has been evolving during the eight years of management. However there has been no substantial change in terms of the objectives and scope of the specific tasks, as well as the overall focus, which has always been on issuing a professional opinion on the aspects that have fiduciary consequences. The third party verifier's views are presented as a report package that includes, for each Province, an Opinion Report and an Internal Verification Report. The first one focuses on the assessment of the most relevant procedures in terms of fiduciary impact and includes the findings and recommendations of the analysis of: (i) the beneficiary’s roster, (ii) tracers’ target achievement, (iii) capitation payments to Provinces based on (i) and (ii), and (iv) use of funds by the Provinces. The second one is a more detailed analysis of the said issues and it includes a review of the supervision environment, the structure and operation of the UGSP and other aspects of compliance of the project. Tasks are executed partially at headquarters and more substantially in the field. One aspect of the responsibilities of the third party verifier that is perceived as a challenge for the authorities of the Plan is the size, distribution and method of selection of the samples for field verification of the Tracers. The authorities of the Plan would prefer that the third party verifier generates a large and widely distributed sample of providers, for example by ensuring that in each Province all the providers are visited. The firms design their samples in accordance with their own criteria and evaluation of risks, and they look for the level of representation that will allow them to issue and sustain an opinion required by the ToR while containing costs and limiting logistical challenges. This issue can be analyzed from different angles:  The objectives and scope of the tasks: a larger sample (in terms of territorial coverage) would likely not have a significant impact on the opinions issued by the auditors. Nonetheless, it would probably have an impact on the management and governance of the Plan for several reasons: (i) there would be more detailed information at the micro level; (ii) it expands the scope of the supervision in the network of providers; and (iii) it supports Provinces in performing the audit tasks (which is their responsibility).  The cost of services: Obviously a larger and more dispersed sample would increase costs, and for this reason there should be a cost-benefit analysis before the decision to expand the objectives of the third party verifier is made. Based on the information from the third party verifier for phase two in 2011, this study has determined that: (i) audit visits were made to 10 percent of providers who reported an average of 1.5 visits per site per year; (ii) distance covered during these visits is about 10 percent of what would be covered while visiting all the sites once in one year. To address this tension between what is adequate to produce a professional opinion and what is desired for supervisory and capacity building information, several solutions are possible depending on where the Plan aims to be positioned along the spectrum of control or fiduciary responsibility on the one hand, and supervision and system strengthening support on the other hand. Proposed solutions are the following: (i) there should be a change in the ToR of the third party verifier to specify sampling requirements, or (ii) at least a portion of the third party verifier contract should include a variable fee adjustable for different sample sizes, or (iii) the central government should contract a different type of firm, with a health background and focus rather than a traditional financial audit focus, that might be better able to adjust to the concurrent supervisory aspects of the contract. In either case, it might be appropriate to evaluate another approach to the problem: since the performance verification is an inherent responsibility of the 38 Provinces, one way to strengthen their role would be to let the third party verification firm oversee the Auditing Plan developed by each Province and then let the Provinces carry out the work. Costs associated with the third party verifier are not inconsequential compared to the capitation payments (6.69 percent). However, this analysis should take into account externalities that have occurred, such as improvement in clinic records, the availability of verified data, and the enhancement of a culture of oversight. The average cost of the third party verifier is approximately $ 12,332 per Province, per month, and it is estimated that the working hours fall between 310 and 350 hours per Province and per month. It is also estimated that the reimbursable costs (freight and transportation) represent between 12 and 15 percent of the contracts amount. The possible ways of reducing the costs are the following:  The limitation of the scope of work of the third party verifier . One possibility is for the audit to focus exclusively on aspects related to the transfer of funds, that is, results indicators. This would mean that the third party verifier would have to provide only the results of the so-called Opinion Report, and leave out the Internal Verification Report. The arguments in favor are the following: (i) verification of services does not generate significant financial impact and responds to a supervision responsibility of the Province; (ii) some tests carried out by the third party verifier in the Provinces relate to activities that repeat every two months and do not result in large adjustments. The information is generally used by the UEC for supervisory functions in its interactions with the Provinces and the tasks may be better suited to one of the departments of the UEC.  The simplification and updating of the standards to concentrate on certain key accomplishments. The rules still include certain requirements (reflected in the ToR and reports of the third party verifier) that were relevant at the time of early implementation of the Plan (for example, some aspects of the compliance with the agreed framework). These requirements may no longer be necessary.  Reducing the requirements pertaining to the qualifications of auditors. Strictly speaking, the task of auditing does not necessarily require the presence of a medical doctor, as it is about data quality checks and record quality. Other specialists could perform this task. Allowing flexibility in hiring could allow firms to more efficiently choose who is needed to complete tasks. However, it should be noted that, according to the personnel of the consulted companies, relaxing requirements may not be helpful or cheaper. That said, there is a strong moral imperative to ensure patient confidentiality when conducting verification involving either patient contact or review of patient records.  Further development of information systems. As the information systems mature and develop, the monitoring and supervision activities change their orientation, scope, and possible cost. The strengthening of information systems related to clinical information in specific Provinces and management processes of the Plan is one of the biggest challenges to be faced in the future that will certainly affect verification practices. 39 REFERENCES Argentina National Ministry of Health, Plan Nacer. 2005-2012.‘Disposiciones Internas.’ 2006-2012. ‘Informe de Gestión del Plan Nacer _____. 2005-2012. ‘Informes de ACE Bimestrales y Cuatrimestrales de Horwath, Price Waterhouse, Bertora’. _____. 2008. ‘Manual de Trazadoras’. _____. 2008. “Cambio en el Modelo de Gestión de Trazadoras.” Serie de Documentos Técnicos nº 31. _____. 2011. “Estudio de Financiamiento y Uso de Recursos de Efectores del Plan Nacer en las Provincias de la Fase I’. This Study has Been Carried Out in Conjunction with Universidad I- Salud, I Gestió and Analogías Consultants. Friedman, Viviana, I. 2008. ‘Taking Stock of Performance/Output Based Financing (POBF) in Health in Argentina.’ World Bank. OLACEFS (Organisation of Latin American and Caribbean Supreme Audit Institutions). 2008. ‘Auditoría de los Programas Sociales, El Salvador’. Perazzo, Alfredo, 2008. ‘Financiamiento Sujeto a Resultados: El Caso Del Plan Nacer’. Roberts, Marc, J., Hsiao, William, Berman, Peter and Michael R. Reich. 2008. ‘Getting Health Reform Right – A Guide to Improving Performance and Equity.’ Oxford University Press. Valle, Walter; Alfredo, Perazzo and Alberto Porto. 2006. ‘Transferencias Intergubernamentales del Plan Nacer como un cambio en el Sistema de Gestión’. Anales de las XXIX Jornadas de Finanzas Públicas, Córdoba. Valle, Walter., Alfredo, Perazzo and Martín Sabignoso. 2008. ‘Los Principios y Valores en el Plan Nacer’. Valle, Walter and Alfredo Perazzo. 2006. ‘Costo Efectividad e Impacto de Inversión en Proyectos Sociales: Algunas Reflexiones y Experiencias.’ World Bank. 2004. ‘Argentina Provincial Maternal-Child Health Investment Project – Adaptable Program Loans I &II. Loan Agreements – Loan 7225-AR.’ _____. 2004. ‘Argentine Provincial Maternal-Child Health Investment Projects - APL I & II – Project Appraisal Documents- Report No. 37702-AR’. _____. 2004. ‘Plan Nacer, Argentine PMCHIP Operational Manual’. World Bank. _____. 2006. ‘Plan Nacer, Argentine PMCHIP Operational Manual’. World Bank. _____. 2007. ‘Plan Nacer, APL I - Mid Term Review – May 2007’. World Bank. 40 ANNEX 1: ORGANIZATIONAL STRUCTURE OF THE UEC IN 2011 Source: Argentina National Ministry of Health, Plan Nacer Note: The organizational structure of the UEC was changed when Programa SUMAR was launched. 41 ANNEX 2: VERIFICATION TASKS OF THE THIRD PARTY VERIFIER RELATED TO BENEFICIARY ENROLLMENT At Central Office Level At Province Levels  Verify that the UEC has cross-checked the Register of Eligible Selection of a random sample of Beneficiaries Enrolled with the OSP and the OSN15. Cross checking is cases from the list of beneficiaries done using computer aided tools (CAAT), for the purpose of identifying: submitted by the UGSP for the o that the procedure has been performed by the UGSP month analyzed, in order to verify o that the registry has been purged of the beneficiaries who have the existence of the enrollment other health coverage (that is, are in PUCO) that should be form, the accurateness of the data removed from the calculation of the monthly capitation uploaded into the system, the payment (the 60 percent) and to whom the provision of services procedure of registration and through the Provincial-Maternal Child Health Insurance SMIP completion of Enrollment forms. should be ceased.  Verify that the Register of Beneficiaries enrolled is accurate: verify, by Also for the enrollment forms that using computer aided tools (CAAT), validating the consolidation pattern: have not been found, it is verified o There are no repeated active beneficiaries whether the form has been o Data is correct and valid requested in prior periods and to o There are no beneficiaries rejected by UGSP without confirm that the absence of the data justification, is due to delayed arrival of the o There has been a monthly automatic removal of children over 6 goods at the UGSP. years of age and pregnant women who have passed the 45 days of termination of pregnancy or for whom two months have Based on the enrollment forms passed from the probable delivery date (FPP) without clinical found, selection of a sample of developments households to verify the existence o Beneficiaries of category 3 and 4 (children of less than 1 year of the beneficiary and the personal old and of 2-6 years old) have at least the ID number of the data. responsible adult (mother, father or guardian) o Beneficiaries of categories 3 and 4 children registered with the Note: ID of an adult, are not repeated in the further data i) the TOR does not require a o No beneficiaries are included in the PUCO Archive representative sample;  Consolidate the registers from all the provinces to confirm that there are ii) the third party verifier usually no duplicated beneficiaries designs a small sample selected by  Verify the process of automatic and occasional cleaning; verify that the geographical proximity UGSP complied with the removal/cleaning procedures by using the iii) difficulties exist in identifying evidence from the CAAT tools to identify: the precise address of the o Beneficiaries that were included in the OSP registers of the beneficiary; previous month and in the OSN registers, o That there has been a monthly automatic cleaning of children under 6 and pregnant women that have reached 45 days after delivery or that have reached 2 months after the probable delivery date, in accordance with the information included in the register of eligible beneficiaries  To ensure the security of the database with regard to the levels of users, permits, access to database administration and applications.  Confirm the correctness of the Back-up procedures.  Check the Control of changes in the original Data Base registers and in the informatics program  Verify that cleaning programs adjust for relevant developments in the census.  Verification of the consistency between the PUCO and the registry of the OSN and the OSP for respective periods 15 OSN refers to National Health Organizations that give coverage to their associates (usually are managed by unions). OSP refers to Provincial Health Organizations that are managed by Provincial Government. 42 ANNEX 3: THIRD PARTY VERIFIER OPERATIONAL VERIFICATION DETAILS Verification topic Processes Details Verification of Validation the compliance of the formal with the aspects Agreement Evaluation  That the Province has created the Maternal and Child Health Insurance between the State of the Provincial by issuing specific rules and regulations, and the adhering obligations  That the Provincial Team Purchase Health Services is functional and has the Provinces of the resources required for normal operation. Province the assembly and maintenance of an organizational structure, in which each staff person contracted possesses the No Objection from the ENCSS and from the BIRF, and a structure proportional to meet the implementation needs of the SMIP and includes the officials listed in Appendix A of the Framework Convention.  That the Province has managed to maintain Provincial Legislature spending allocated to health, and there is evidence of commitment to manage the resources needed to maintain the program at the provincial level.  • Verify that the province has opened an SMIP account for use under the rules of the program and that it is coincident with the UEC.  That the funds are used to respond only to the purposes specified in the Program.  Verify that, upon signing the Convention, have signed the Letter of Intent.  Verify that the agreements signed between the federal government and the provinces are Adhering to the Framework Convention Model designed by UEC.  Enable the Province to act as centers of Registration Agents for beneficiaries, to enroll only the eligible population and that the processes and procedures used to ensure this inscription are forming a single register of eligible beneficiaries enrolled.  To be provided to eligible beneficiaries enrolled with credential of SMIP.  Sign contracts only with authorized providers.  Mechanisms to implement hiring and pay are according to the regulations.  Meets the delivery on time and the patterns of beneficiaries are properly purged, the pattern of the OSP, the information necessary to validate all transfers.  Create and maintain a database of qualified providers.  Report annually to the nation's annual work program containing: Actions planned for next year, given the Nomenclature values, goals that are expected to fulfill in respect of the tracers, the enrollment rate of the planned investment plan, technical assistance plan and a training plan. The work program once accepted and signed by both parties will become the annual commitment.  Facilitate the tasks of monitoring, auditing, monitoring and evaluation of the program.  Use the basic software solution provided by the Nation.  Carry out an extensive campaign of advertising and communications oriented program eligible population and considering the native populations, given to minorities with disabilities such as deaf, blind, etc.  Mechanisms are implemented with public participation for the purpose of evaluating the effectiveness and efficiency of the program. 43 Verification topic Processes Details Evaluation  Transfer an amount calculated on the basis of eligible beneficiaries enrolled of the in decreasing order, to finance the purchase of services that have been made obligations since the UGSP and have been used to perform the Nomenclature. of the State  Provide a solution to the Province’s basic computing to the management and administration of the SMIP.  Check the formal aspects of the agreements signed and the subscription and involve IBRD for No Objection.  Verify that the Provinces Adhering keep validating their status.  Verify that the project is in effect.  Verify that it has not been more than three months after the expiration of the annual commitment and without the signing of a new one Maintain the UGSP operational.  Verify that no serious failures have occurred by the Province. Verifying that the  Check the delivery time by: verifying that within the first 10 days of each respective month, the Province sends the liquidation form and request for transfer with province is an affidavit along with an updated Register of Eligible Enrolled Beneficiaries submitting an and an updated OSP registry. These registries should be updated on the last acceptable OSP day of the previous month. registry to the  Verify that the information is filed electronically and accompanied by a letter UEC in an affidavit signed by the highest competent authority, endorsing the validity of its content or any alternative method proposed by the Unit to validate data integrity. Verifying the  Verify the existence of contracts with Authorized Providers: verify that existence of the providers have signed contracts Enabled with USGP and that they meet the contracts between objectives, policies and standards set by the program, and that it establishes UGSP and health rights and obligations consistent with the objectives of the project. facilities, and that  Ensure that contracts are referred to the practice and the prices established in they are in line the Unified Nomenclature for the Province to hire the Lender. with the Verify that formalities are required to consider the contract as valid guidelines of the (signatures, authorizations, etc.). Program  Check that the text fits contracts project guidelines and standards established by the UEC: to validate which of the contracts do the following: o Explicitly include a commitment to comply with the rules of the program, o Incorporate the requirement to have basic information and updates of medical records, o Include the acceptance of supervision, monitoring and auditing by the UEC or its designees, o Consign to each beneficiary and maintain a medical record following the rules of the Ministry of Health of the Nation, o Undertake to use the forms that meet the minimum requirements of the program for cases of psychiatric liaison, referrals, patient records, birth / postpartum and perinatal first consultation.  Verify that the payment mechanisms established in the contracts are how the financial transfer mechanism is used in the SMIP Account for the payment of services. 44 Verification topic Processes Details Verifying the  Verify that UGSP sends the following to UEC within the first 10 days of consistency of the each month: billing sent by the o Form of liquidation and transfer capitated request, Provinces to UEC o Magnetic media copies of: a) the register of eligible beneficiaries justifying the enrolled to the date of the last day of the previous month, and b) monthly transfer the census date of the OSP, (60 percent of o Surrender of expenditures from the account during the previous month “capitation”) SMIP, both in electronic and printed form, initialized by the highest competent authority of UGSP, o Last bank statement of the provincial special account SMIP with maximum age of two months.  Check that the number of entries in the Register of Registered Eligible Beneficiaries of SMIP is coincident with the number of beneficiaries billed to UEC: Once all the checks on the standards detailed beneficiaries are deemed liquidated by the Provinces and compare them to make sure they are adhering with the Register of SMIP. It shall inform the UEC debits to be made on the settlements.  Check the consistency and timeliness of debits made to the provinces with the errors found in the proceedings.  Verify that clearances meet the per capita formula based value ($ 17) x Beneficiaries Eligible Enrollees.  Check that the transfers take place only towards the SMIP Accounts opened for this purpose in UGSP. Verifying the  Verify compliance with the manner and time of payment to ensure accuracy, compliance with the conditions laid down in the Framework Agreement timeliness, and between the federal government and the provinces attached, and verify that compliance with the transfers are made to the accounts of SMIP open in each of the the terms of the jurisdictions, that they are made within the prescribed period and settled for Program by UEC amounts less debits corresponding to each period. in regard to capitation payments to the provinces Source: Bimonthly Third Party Verifier Internal Verification Report to UEC 45 Verification in results-based financing (RBF) mechanisms is one of the key differentiators between it and related health financing structures such as social health insurance. Verifying that providers have achieved reported performance in RBF mechanisms is considered a crucial part of program implementation and key to maintaining trust through transparency, as well as the viability of the mechanism. Verification is however a process which has thus far been little studied. Information on the methodologies used in different settings (including frequency and sampling methodology), the effectiveness of the verification process, the direct and tangential effects, and the cost is scarce. Plan Nacer employs one of the largest RBF mechanisms in the world and is therefore an excellent case study for the role, methodology and effects of the verification process. This study will give the background to Plan Nacer, detail the major characteristics of the verification process and draw lessons on the process which can inform the design of verification in RBF mechanisms in other countries. ABOUT THIS SERIES: This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author/s whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Editor Martin Lutalo (mlutalo@ worldbank.org) or HNP Advisory Service (healthpop@worldbank.org, tel 202 473-2256). For more information, see also www.worldbank.org/hnppublications. 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org