Impact Evaluation Design for the Zambia Access to ACT Initiative (ZAAI) – A Project Concept Note Improving the Public Sector Supply Chain, Community Engagement, and Affordable Access to ACTs in the Private Sector I. POLICY CONTEXT – CURRENT SYSTEM AND MOTIVATION FOR INTERVENTION The Zambian National Malaria Strategy has achieved remarkable success in expanding access to preventative services. With the most recent rollout of 3 million long lasting insecticide treated nets to date, ITN coverage has increased to a national average of 60% (MOH 2008). Indoor residual spraying (IRS) is planned to be scaled up from the current 15 districts to 36 over the coming year. Zambia is developing an innovative model for community engagement by strengthening its Community Health Worker network and improving their skills with regard to malaria prevention and control activities. Despite the success in malaria control the Zambian burden remains widespread, with an estimated 3.7 million malaria cases of P. falciparum and 6,500 malaria-related deaths in 2006 (source?). One critical area of disease prevention and control where less progress has been observed is case management, especially in rural areas. According to the preliminary results from 2007 DHS, only 24% of urban children and 19% of rural children under 5 received malaria medication within 24 hours of onset of fever. In related analysis, the 2006 MIS finds that only 8% of children under-five received Coartem within 24 hours even though ACT therapy is now the first line treatment and is available free of charge at the point of service in the public sector as well as sold in selected private sector outlets. This relative underperformance in case management is due to several factors including: • Frequent stock-outs in public health facilities have hampered access – the 2006 Pubic Expenditure Tracking Study reported 81% of all health facilities experienced ACT outage at some point in the last three months. • In the private sector both the high price of ACTs and general lack of awareness constitutes a barrier to access – a recent University of Zambia/Clinton Foundation study finds the sale of significantly lower priced Chloroquine and SP/Fansiadar to be much more common in private outlets than ACT sales. The 2008 Annual Malaria Plan identified a number of bottlenecks where successful interventions can enhance the access and affordability of ACTs in both the public and private sectors. A general consensus among the National Malaria Control Center, the Ministry of Health, and key donors has emerged to address these concerns. This consensus holds that finding solutions for the described bottlenecks in the two sector can be best achieved by implementing complementary intervention strategies in both the public and private sector to increase ACT access as listed below: 1. Through the Public Sector: Improving the Supply Chain of Essential Medicines from Central Stores Down to Primary Care Facilities 2. Through the Private Sector: Improving Access to and Affordability of ACTs with the Targeted Subsidization of ACTs to Private Sector Outlets The objective of the proposed impact evaluation is to measure and provide rigorous quantitative evidence on the effectiveness and cost effectiveness of each pilot intervention. The paired public and private sector interventions presents a unique opportunity to measure and evaluate concurrent improvement in both public and private sector accessibility and their joint effect on household access to first line treatment. The technical design will be as thorough as the context allows in order to assure the internal validity of the findings. Simultaneously the design maximizes the external validity of findings, and thus informs scalable learning, through the implementation of the interventions in a variety of settings. The effectiveness of the proposed interventions is anticipated for the following reasons: • Improving the public sector supply chain management will reduce stock-outs in public hospitals and health facilities. With improved stocks levels, the population will be able to better rely on public services/ACT delivery. Improved services at the point of delivery should increase utilization and the cost-effectiveness of service provision. The positive perception by people visiting public facilities and receiving timely treatment should also influence the decision of other households. • Due to the coincident introduction of an ACT subsidy and diagnostics in the private sector, as well as sensitization efforts directed towards retailers, malaria- related private sector health seeking behavior is expected to change. The objective of the subsidy is to reduce the market price of ACT to a level that is within the willingness to pay (WTP) bounds of the majority of the poor population, to achieve price competitiveness with mono-therapies and non-optimal drugs so that the clinically more efficacious treatment will crowd out demand for inferior malaria therapies/drugs, and to sensitize consumers and private drug suppliers on the need for diagnostics when accessing such effective anti-malarial medication. • Observing the impact of these two interventions when implemented either separately or conjointly is a unique opportunity to inform policy decisions regarding the accessibility and affordability of anti-malaria drugs accessed either through the public or the private sector. The proposed study will quantify the relative effect of each intervention, as well as their combined effect on household decision making on malaria treatment. II. DESCRIPTION OF INTERVENTIONS 1. Public Sector Intervention In May 2007, the World Bank conducted an assessment of the public-sector supply chain for essential medicines in Zambia (Kim O. Beer, 2007). This, as well as studies by Yadav (2007) reiterate that there is poor visibility of demand and inventory information and lack of accountability in the downstream supply chain (i.e. in the secondary distribution from the districts to the health centers/posts) (Figure 1). Currently Medical Stores Ltd. (MSL) is responsible for the primary distribution of drugs, from the central warehouse to district stores, and the secondary distribution is the responsibility of the districts themselves. The WB report hypothesized that expanding the mandate of MSL to include secondary distribution may improve the system performance. To address these problems and ensure adequate stocking at the facility level, experts proposed a number of intervention pilots to redesign the distribution system. There are two primary channels for system innovation. These include: i. Through resource reconfiguration in the existing distribution system (e.g. allocating more planning capacity to districts; allocating or creating new transport 2 resources at the district). Operationally this can be achieved through the introduction of a Commodity Planner (CP) at the District Level who is responsible for stock management, coordinating information from Health Facilities, providing technical assistance to enhance stock flows and obtain improved and timely data from health facilities (HFs). ii. Through change in the physical and operating structure of the distribution network, in particular the introduction of a cross-docking system that eliminates the need to hold stores at the district level. This design, in addition to the improved flow of stocks, is expected to reduce the scope for leakages out of the public system. Selection of Pilot Interventions Following technical proposals by logistic experts and discussions with the World Bank and JSI concerning technical and operational merits (i.e. opportunities for scalable learning, sustainability, expected impact on system, etc.), the MOH and MSL agreed to focus on testing the following three interventions in the public sector supply chain: a. District Store maintained with Commodity Planner- System A This system represents the smallest incremental change from the existing arrangements. The districts store is maintained as a stockholding point, and orders monthly on MSL. MSL will make monthly deliveries to the districts in which this model is piloted. Each order from the district store to MSL will be based on aggregated consumption data collected from all health centres in the districts. The health centres in the districts will report to their district monthly, and will receive one or two deliveries per month from the district (supported by the commodity planner) dependent on the availability of storage space at the health centre and a vehicle and driver at the district level. A commodity planner, employed by MSL, will be placed at the district store to support all supply activities. The purpose of system A is to evaluate the impact of the introduction of the Commodity Planner and the new ordering/reporting system on the availability of drugs and medical supplies at service delivery points. For a diagram representing system A, please see Appendix Figure 1. b. District Store converted to a Cross-Dock, no Commodity Planner System-B This system converts district stores to cross-docking points. No stock will be held at the store; rather, it will act as a pass through for consignments already packed and labeled for health centres. The district cross-dock will collect reports monthly from all health centres, convert them into orders and forward them to MSL. MSL will pick and pack orders for each health centre, and deliver to the district cross-dock once per month. The district cross-dock will then deliver once or twice a month down to each health centre, based on the availability of space at health centres and vehicles at the district level. No commodity planner is proposed for system B. This system seeks to evaluate how well the cross-docking system can operate with no extra resources provided at the district level. The purpose of system B is to evaluate the impact of converting district stores to cross- docking points (with MSL packing for individual health centres), and introducing a new 3 ordering/reporting system, on the availability of drugs and medical supplies at service delivery points. For a diagram representing system B, please see Appendix Figure 2. c. District Store converted to a Cross-Dock, with a Commodity Planner- System C This system has the same features as system B, with a commodity planner added. The purpose of system C is to evaluate the incremental benefit, if any, of adding a commodity planner at district level in a cross-docking system. For a diagram representing system c, please see Appendix Figure 3. Scope of pilot: Each of the three systems will be implemented in six districts representing a range of administrative, socio-economic, and epidemiologic conditions found in rural and peri- urban Zambia. 2. Private Sector Intervention Private Sector Subsidy to Promote ACT as First Line Malaria Treatment Over the last two decades, parasite resistance to common malarial treatments such as Chloroquine and Sulfadoxine-Pyrimethamine (SP) has spread. In response, most national governments have adopted Artemisinin-based Combination Therapies (ACTs) as first-line treatment for uncomplicated malaria, in line with the recommendation of the World Health Organization. While the efforts of national government programs and international donors have increased ACT access through public health systems, the high cost of ACTs has limited their uptake through the private sector. To address this situation and to mitigate the risk of development of resistance to ACTs, the Institute of Medicine recommended the establishment of an international financing mechanism – now known as the Affordable Medicines Facility – malaria (AMFm) – to lower the price of ACTs to both the private and public sectors through a subsidy applied at the global level (Arrow et al, 2004). The role of the private sector in malaria case management in Zambia is significant. As captured in Table 1, roughly 42% of the population that seek care for fever do so in the private sector (LCMS 2004). Furthermore, private sector care is not merely the domain of wealthy households. Roughly equal proportions of individuals in each expenditure quintile access anti- malarials in from private outlets. Table 1. Percent of individuals seeking treatment in the non-public sector for suspected fever/malaria Expenditure quintile Total Age category and data source Urban Rural Bottom 2nd 3rd 4th Top N All individuals, 2004 LCMS 42.0% 35.4% 36.9% 41.4% 36.8% 32.2% 45.5% 3203 Children under 7, 2004 LCMS 38.7% 36.7% 34.3% 50.5% 49.2% 23.6% 35.8% 316 Children under 5, 2006 MIS ---------- ---------- ---------- 16.4% ---------- ---------- ---------- 360 Note: Results for 2006 MIS do not distinguish by rural/urban or expenditure quintile. Non-public sector does not include mission and mining facility. While work by Yadav and others provides insight into the top level of the private supply chain for anti-malarials, little is known about the semi-formal and informal retail outlets that 4 provide treatment to patients. To provide some initial evidence, the Clinton Foundation HIV/AIDS Initiative (CHAI) and University of Zambia, with support from the World Bank and National Malaria Control Center, conducted a rapid analysis of private retail and wholesale outlets in six districts of the country to guide ongoing deliberations on a national ACT subsidy through the private sector. This recent small-scale study by the CHAI and University of Zambia (2008) shows that the majority of medicines available and purchased in the private sector are Chloroquine and SP. Artemisinin mono-therapy (AMT) is also available for sale. The largest barriers to ACT utilization appear to be: i) Stocking: In all surveyed districts, SP has the highest penetration among private sector outlets stocking anti-malarials. ii) Price and Business Practice: The high price of ACTs1 is a principal factor limiting the private sector uptake – the continued prescription of SP from public facilities also appears to contribute. iii) Perception of Demand: Private outlets perceive the demand for ACTs to be low Shop keepers cite price as the primary factor for not stocking ACTs while low demand and lack of information appear to limit ACT and RDT uptake. Given the high price and lack of awareness among both consumers and retailers, this study proposes to explore the effect of a price subsidy, incentives for consistent mark-ups to assure low prices for the subsidized product, and training for shop keepers to alter stocking and dispensing behavior. The declining occurrence of malaria incidence in Zambia due to successful prevention interventions also creates an urgent need for proper diagnosis of all febrile cases as it is no longer possible to consider any fever case to be malaria. The national malaria control program has started deploying rapid diagnostic tests (RDTs) in the formal healthcare system as part of its case management and diagnosis strategy. However, diagnosis capacity is almost non-existent in the private sector. In order to ensure that those seeking treatment outside of the public health care system are also incentivized and educated about use of RDTs, this private component will therefore subsidize both RDTs and ACTs in the private sector. Subsidy Design The private sector component has two principal components: (i) Subsidization and Distribution: The ACTs and RDTs are procured from the manufacture at its public sector price and then sold at a subsidized price to pharmaceutical wholesalers, which deliver the products, through their normal distribution channels, to eligible drug outlets in the designated districts included in the pilot. (ii) Implementation of supporting activities: A number of supporting interventions (e.g. repackaging, determining a suggested retail price, public awareness campaigns, incentives to wholesalers and training to drug dispensers) will be implemented. These interventions seek to increase demand for the subsidized ACTs and improve the quality of care that patients receive in the private sector. 1 Overall, ACTs and Artemisinin monotherapies (AMTs) were the most expensive anti-malarials observed in sampled outlets at average reported rates of $8.48 and $7.92, respectively. Chloroquine and SP were the least expensive anti-malarials observed, with average retail prices of $0.38 and $0.59. Anti-malarial retail prices varied across and within districts. The greatest variations were witnessed in ACT pricing, with, on average, outlets in urban areas charging much higher prices ($10.58 per course) than those in rural areas ($3.53-3.70 per course). 5 Key aspects of the private sector pilot include: SELECTION OF OUTLETS FOR THE PILOT The pilot will aim to maximize access to ACTs for all socio-economic groups of the population. In order to ensure equitable access, the selection of outlets for the pilot will be based on analysis of the type and number of existing outlets in each area. The criterion for selection will ensure that communities with a high malaria burden with poor access to ACTs will particularly benefit from the pilot. However, certain basic conditions on drug storage, dispensing, willingness to comply with contractual obligations of the pilot, willingness to undergo training etc. will be made important pre-conditions for outlets to be eligible to stock the subsidized ACTs. INCENTIVES FOR USE OF RDTS There is evidence that RDT results are often ignored when treatment decisions are made. To address this issue, the project will include strong communication components to educate the community on the importance of diagnosis. In addition, the pricing of the RDT will be such that the cost of diagnosis to the end-patient is minimal or zero. A willingness-to-pay study will be conducted to assess the appropriate price for RDTs. Adequate management algorithms will be included for management of negative cases (non-malarial febrile illness) and the dispensing outlets will be trained to refer patients to the nearest public health facility. To ensure that the wholesaler and the dispensing outlets have adequate incentives for dispensing ACTs only to malaria positive patients, the margins and incentives for them to dispense RDTs will be set a little higher than the margin to dispense ACTs. PILOT DISTRICTS The pilot project will be implemented in four highly endemic districts that are representative of peri-urban and rural Zambia in terms of socioeconomic status, access to public facilities, presence of private outlets and malaria. CONSUMER AWARENESS AND BCC Activities will be undertaken to raise public awareness about the need for rapid treatment of malaria with ACTs; the availability of affordable ACTs and RDTs at the selected outlets in their communities; and the need for proper diagnosis using RDTs. Multiple approaches including print, mass media, and community outreach will be used. These activities will be combined with NMCCs existing communication and BCC/IEC strategy to convey other key malaria messages. SUGGESTED RETAIL PRICE An appropriately-determined suggested retail price (SRP) will be marked on the packaging. Once again, these prices will be set to achieve the lowest possible prices for end patient while maintaining a healthy incentive for the wholesalers and dispensing outlets to continue distributing the product efficiently. INCENTIVES FOR WHOLESALERS Experience from pilot projects in other countries4 suggests that the private sector distribution of subsidized ACTs to peripheral areas is successful only when there are adequate incentives for the wholesalers and retailers. The Government of Tanzania and Clinton Foundation ACT subsidy pilot in two rural districts has found that only 38% of stores in remote areas versus 80 percent of stores in more densely populated areas were stocking ACTs five months after the subsidized drugs were available in the market.2 Distribution to more peripheral areas is limited due to: (i) 2 Government of Tanzania-Clinton Foundation. “Tanzania Pilot ACT Subsidy: Report on Findings.” April 2008. 6 long distances and poor road conditions to peripheral areas; (ii) limited capital of retail outlets to purchase when the distributor delivers; (iii) lower demand in rural areas. Thus, the pilot project will include a system of incentives where the wholesalers/distributors will receive performance awards upon providing evidence that they are distributing subsidized ACTs and RDTs in the selected peripheral dispensing outlets in addition to the easier to reach ones. 3. Interaction between the Public and Private Sector Interventions Finally, the study proposes to overlay the public and private sector intervention districts in order to measure the joint effects of the two simultaneous interventions. Based on the descriptions thus far on the public and private sector interventions, the treatment and control arms are as follows: Control: Current State (Neither public nor private sector intervention introduced) Treatment 1: Public Sector Intervention Only a) Enhanced Current System b) Cross Dock only c) Enhanced Cross Dock Treatment 2: Private Sector Intervention Only Treatment 3: Interaction of Public and Private Sector Interventions (Combined Effect) 4. Complimentary Interventions Where possible, the research will take into account the role of complementary demand side interventions, such as the role of Community Health Workers (CHW). The ongoing CHW pilot in Livingstone and the plans for scaling up the use of CHWs into further 9 districts presents an opportunity to measure the combined effect of the public and private interventions in an environment where Zambia is introducing Home Management of Malaria (HMM). This effort includes including ACTs in the CHW Kits, as well as training CHWs to be able to promote and effectively and safely dispense ACTs in their communities. HMM is especially important for rural, hard-to-reach areas, where households have limited access to facility-based care. The CHW intervention is planned to be scaled up to ten districts by the end of 2008. Upon successful piloting in 2008, the CHW program would be expanded to 19 district, then to 27 districts, and, as resources allow, country-wide, to all 72 districts. III. PRIMARY RESEARCH QUESTIONS Research Questions on the Public Sector Intervention to Improve the Public Supply Chain 1. How does strengthening the supply chain in the public sector affect priority health outcomes through the increased availability of essential drugs, including ACTs, for households? 2. Which one, if any, of the supply chain pilots is the most cost effective in ensuring adequate supplies of essential medicines at the facility and community level? Research Questions on the Private Sector Intervention 7 1. How does a flat subsidy to the manufacturer for both diagnostics and curatives, coupled with BCC activities and community outreach, affect household demand for ACTs in private sector outlets? 2. What is the pass-through rate of subsidy to the end user? 3. To what extent do households switch from conventional malaria drugs (CQ, SP, AMTs) to ACTs as ACT price is lowered to a more competitive level? Research Question on Public and Private Sector Interaction 1. Under the pilot interventions, what is the extent of switches in health demand from public to private sector, or from private to public sector, for malaria related treatment seeking behavior? What are the key characteristics driving choice of care between hte public (including CHWs) and private sector? 2. Which of the three interventions (i) public supply chain, (ii) private subsidy, or (iii) both is the most cost effective for achieving priority health outcomes? IV. DATA SOURCES AND KEY OUTCOME MEASURES As the proposed design will inform GRZ on a variety of factors, study information will be collected across a range of topics stretching from process inputs to ultimate health outcomes. It is important for key outcome measures to capture: i) The extent of the effect of the public interventions on ACT and other essential medicine stock levels at health facilities, household health seeking behavior in regards to malaria and other illnesses, the consequent impact on HH health and socio-economic outcomes, as well as the public sector cost of such an improvement. ii) The extent of the effect of the private intervention on household behavior, health, and socio-economic outcomes as well as the cost of attaining such an improvement. iii) Whether substitution behavior from the public to the private sector occurs, or from the private to the public sector, given the improvements in one or both sectors. Data Sources: The data necessary to capture these measures will be derived from the following main data sources: i) Public Sector Intervention - Public Sector Administrative Data and Facility Surveys a. Health Facility Data through Routine (and recently upgraded) HMIS/MSL Reporting and Upgraded HMIS/MSL b. Dedicated Survey on Public Sector Supply Chain c. Exit interviews from public clinics d. CHW Survey ii) Private Sector Intervention - Private Sector Facility Surveys 8 a. Facility Census of Private Sector Outlets b. Audits, including use of ACT tracing through uniquely labeled packages c. Exit Interviews from private dispensers d. Mystery Shopping iii) Private and Public Sector Interventions In addition to the administrative data and dedicated surveys, the following additional sources will be used to (a) provide a comprehensive picture of health seeking behavior at the household level, (b) analyze the equity implications of each intervention and control for potential confounders, and (c) measure the costs, relative costs and cost effectiveness of the proposed interventions. - Household Survey - Complementary Data o Monthly Weather Statistics o Community Survey - Costing Data Rationale and Plan for Data Sources: - Public Sector Administrative Data and Facility Surveys Regular administrative data on facilities and supply chain management, such as the HMIS and MSL reporting, does not cover all key variables of interest necessary to track and evaluate the described interventions. To complement the existing administrative data streams, the IE team has been collaborating with the MOH, NMCC, MSL, JSI, the University of Zambia, and the Clinton Foundation to upgrade the reporting system and develop the instruments required to capture the effect of the interventions in the public and private sectors. In the public sector, the reform effort on supply chain management will collect a. Administrative Data: Monthly reports and summaries from MSL will be produced to allow tracking of data. Collaborating partners have agreed on the list of tracer products and other data elements to be collected. In addition, the first stage baseline data collected from JSI will be used to monitor the performance in terms of stock outages at the facility level. b. Dedicated Surveys: Key data provided through surveys of public facilities and patients include: i) facility type; ii) access to electricity / water / sanitation / telephone; iii) personnel per capita; iv) HR (absenteeism, ghost workers, staff education, qualified staff) v) working equipment; vi) material (stock of drugs and commodities, stock outs of tracer items, etc.); vii) incentives (compensation, salary retention, delays in payment, supervision and its frequency, public resources share in revenues, public resources per capita, external resources, donor support, user fees, user fee’s share of client income, informal payments); viii) service outputs (number of consultations, etc.); ix) provider KAP for malaria related illness; x) consumer satisfaction (accessibility, affordability, quality, timeliness, information value, etc.). 9 c. Further, for the complementary intervention in the public sector, data on CHWs will be retrieved from routine administrative data, and the upgraded facility and supply chain reporting system. CHW surveys will take place in selected communities. - Private Sector Facility Surveys In the private sector, the ACT subsidy intervention necessitates improved mapping and stock taking of various levels of private outlets, and improving the tracking of drugs, specifically fever/malaria related drug stocks and diagnostic capacity. Data has been scarce in this context. The design has been informed by a small-scale study performed by the Clinton Foundation and the University of Zambia (June 2008). As stated above, the evaluation will require: a. Census of Private Sector Retail Outlets in selected districts b. Audits, including use of ACT tracing through uniquely labeled packages c. Exit Interviews d. Mystery Shopping The (a) census will provide a mapping of the outlets (location, catchment, concentration, accessibility, etc.) as well as information on stocking and prices and the malaria related knowledge of retailers. (b) Audits will enable measuring, for example, the effect of volume incentives by recording sales. (c-d) Exit interviews and mystery shopping will shed light on consumer choice factors, the availability of ACT and diagnostics, etc., and ascertain pricing, price mark-up information. - Household Survey Data Administrative data and facility surveys suffer from well know limitations (O’Donnell et al. 2008), such as i) concerns regarding quality (e.g. reporting system capacity, potential for data tempering); ii) representativeness (biased estimates due to sample selection; e.g. the difference in results/variables for those who seek care at facilities and those who self-treat can be systematic); and iii) little complementary data on important covariates, for example, household socio- economic status, education, health status, labor productivity, etc., as well as the effect of the intervention on these household variables over time. To offset these limitations, the evaluation will include a household survey. A combined malaria indicator and socioeconomic household survey will be administered in randomly selected households in both the control and intervention districts prior to the intervention (baseline data), and one year following the baseline survey (follow up round). • The household survey modules will provide data on: household composition (age, gender, etc.), consumption, assets, education, labor supply, health seeking behavior, fever/malaria episode-related KAP, history of malaria within the household, treatment seeking behavior, WTP for anti-malarials, fever/malaria related expenditures, and opportunity costs of illness. • Biomarker Tests: Parasite prevalence, hemoglobin, and anthropometry will be collected from all household residents. Upon consent from the household member or his/her guardian, parasite prevalence will be tested using rapid diagnostic test kits (RDTs). The procedure is mildly intrusive, whereby a small sample of blood is taken by standard finger-prick methods using a sterile lancet. Trained public health technicians will be responsible for all blood collections. - Complementary Data 10 Complementary data will be collected to track and understand the effect of potential confounders, and to, ideally, ensure lack of contamination between treatments and control groups, or - if unavoidable - to best mitigate these effects during the analytical work. Complementary data includes: i) monthly weather statistics; ii) community factors, including community participation, changes in behavior communication related to fever/malaria prevention and treatment, etc.; and iii) specialized agency consultations (MOH, NMCC, NRA, MLS, DHMT, etc.) to track/control for confounding interventions, such as introduction of new regimes (e.g. RBF, CHW, etc. pilots; additional preventive intervention ITNs, IRS etc. through other donors; changes in the regulatory regime, etc.) - Costing and Cost Effectiveness In addition to tracking operational progress, measuring the effectiveness, and the impact of the interventions on the population, the study includes a rigorous costing and cost effectiveness component. The cost-effectiveness analysis (CEA) will provide evidence on the relative costs and consequences of different interventions in order to assist in priority-setting and budget allocation. Costing will inform on accounting and economic costs of the interventions. Cost effectiveness will inform on the gross (incremental cost of intervention only) and net costs (incorporating potential cost savings as a result of the intervention, measured e.g. as cases averted, reduction in productivity loss, etc.) of the interventions. Specifically, ƒ Cost data will be analyzed to assess the unit and incremental costs of the alternative supply chain management modalities (public sector intervention), and the ACT and diagnostic subsidy (private sector intervention). These will be compared to the baseline case of the existing system and alternative treatment regimes. ƒ An ingredients approach will be used to calculate unit costs. Resource categories will be assessed independently to identify the more costly components to running the given public supply chain modality, and treatment regime. ƒ The primary resource categories include: personnel, equipment and supplies (e.g., ACT, RDT, etc.), vehicles and transportation, training, community sensitization, initial capital investments (for buildings, hiring staff), and overhead. Identifying the relative contribution of resource categories to the total cost of each intervention is useful for improving resource allocation decisions and overall program efficiency. Both financial (e.g. reflecting actual market prices) and economic costs (reflecting the true opportunity costs of donated inputs and volunteer time) will be analyzed and compared to assess issues of affordability and sustainability. All cost analyses will be divided to illustrate the costs incurred at the start of the project versus maintenance costs after program has been operational for a period of time. The analysis will emphasize the provider perspective, but we will also assess the direct (e.g., savings to individuals and households from medical treatment averted) and indirect costs (e.g., the value of lost work time from illness and care-seeking) to individuals and households. Indicators: Through the listed data sources numerous indicators will be collected. For the purpose of study design, the main indicators considered will be the direct targets of the pilot interventions: 11 1. Percentage of Public Facility Stock-Outs of Key Commodities (Including ACTs) (Facility Reporting System, MSL-JSI) 2. Percentage of Fever Cases Treated with ACT within 24 Hours (Facility Records/HH Survey) 3. Availability and Price of ACT in the Private Sector (Private Sector Outlet Survey/HH Survey) Additional measures are numerous but important ones include: 1. Parasite Prevalence (HH Survey) 2. Reduction in Malaria Incidence for Children Under 5 (HH Survey) 3. Reduction in Malaria Mortality for Children Under Five (Facility Records/HMIS) 4. Days of school and work lost to illness (HH Survey) 5. Public Sector Supply Chain Reporting (Upgraded System, MSL-JSI) V. IDENTIFICATION STRATEGY Rigorous evaluation demands that any observed change in outcomes in areas that receive treatment must be compared with a valid counterfactual area that represents the course of events that would have occurred in the absence of the intervention. This counterfactual proxies for what would have happened without the program. There are various methodological approaches to the construction of a valid counterfactual, but by far the most rigorous evaluation design is an experimental design where treatment/control status is assigned to a locale on a randomized basis. Randomization assures that all units have an equal chance of control or treatment status and satisfies the conditions of a valid counterfactual comparison, that is (1) all pre-intervention factors/characteristics will be on average equal between groups and (2) the only difference in observed outcomes is due to the intervention and not to other observed or unobserved factors. The public sector interventions proposed here, however, are implemented at the district level. A total of 18 out of 58 peri-urban and rural districts will receive some form of the public sector intervention. Due to the relatively small sampling universe, districts will be randomly selected from strata (defined below) to ensure a balance across treatment arms in regards to district characteristics. A total of 24 districts (including control districts) will be included in the public sector study. Urban districts are not included due to the relatively high performance of the health system in urban areas as well as relatively low malaria burden. 18 of the remaining 58 districts are peri- urban, 40 are rural. Maintaining a relative balance, this implies the selection of 8 peri-urban and 16 rural districts. The treatment arms and control status will be randomly assigned within this sample of 24, stratified by rural or peri-urban status as well as by general region of the country – The Eastern and Northern Region, the Central Region, and the Southern and Western Region. Districts are stratified by region to ensure a geographic balance in the selected districts and to further control for possible region wide influences such as general weather patterns. Facility data collected in 2006 indicates there are 4 significant predictors of ACT stock outage at the facility level besides the, additionally significant, rural/peri-urban status. These predictors are: malaria incidence (a positive relationship – the greater the malaria incidence the more likely a reported ACT stock outage), likelihood of phone at facility level (a negative relationship), district population (negative relationship), and average catchment area of facility 12 (positive relationship). Together these predictors account for about 15% of the variation in observed ACT outages in the non-urban districts. Given these findings, districts within each region and peri-urban or rural category are further grouped into “high risk” districts (those that have either 3 or all 4 of these risk factors) and “low risk” districts (those with 2 or fewer risk factors). An equal proportion of high and low risk districts are randomly selected into the study sample. As a final step, the assignation of districts to interventions 1 (district stores plus commodity planner), 2 (cross-docking), 3 (cross-docking plus commodity planner) or Control (existing system) are also randomly determined. This method results in the selection of districts listed in Annex Table 1. Geographic Targeting of the ACT Subsidy Intervention The ACT subsidy intervention is geographically targeted to four districts. Limiting the pilot’s scale (compared to the public intervention where the number of intervention districts is 24) is required due to the costly nature of the ACT subsidy. Furthermore, due to the cross-over nature of the joint study, two of these districts will also receive some version of the public sector intervention – either district stores plus commodity planner or cross-docking plus commodity planner – as well as the home-based management of malaria. Only rural and peri-urban districts will be randomly selected into the ACT subsidy pilot since the malaria burden in urban districts is comparatively low. Again, the approach is to randomly select districts from within strata. In this case the strata are three-fold: peri-urban or rural location, general region of the country, and public sector pilot treatment status. The four selected private sector districts are listed in Annex Table 2. VI. SAMPLING AND POWER CALCULATION According to the 2006 PETS data, 81% of facilities experienced some outage of ACTs in the 3 moths prior to survey. A study of 6 districts per arm is sufficient to identify a decrease of 40 percentage points in the stock outage rate, at a standard significance level of .05 and a power of .8, given the following assumptions: • 15% of observed variation in facility stock-out rates is attributable to observed district characteristics. • An average of 15 facilities per district. • A hypothesized intra-cluster correlation of .35 (the observed PETS data correlation in .24) If instead we use adopt the observed intra-cluster correlation (.24), then we would be able to identify a 33 percentage point reduction in the stock outage rate at the same levels of precision and power. Distinguishing the relative improvements in stock outages across different models will be more difficult at standard levels of precision if the performances of each pilot intervention are expected to be more similar to each other in relation to the control group. However the each pilot should lead to a great deal of procedural learning and so will help to discern the effectiveness and costs of each intervention. With regard to the household survey, a critical data input to the evaluation of the private sector study as well as the joint impact of both private and public sector intervention in relation to 13 none or either alone, the sample will be composed of households from 8 districts. This includes the two districts that receive both the public and private sector intervention, the additional two that receive the private sector intervention alone, two districts that receive the public sector intervention (one type A district and one type C district in order to match the two cross-over districts), and two controls. The four sampled districts that do not receive the private sector intervention will be purposively chosen to match the four private sector districts in terms of peri- urban or rural location, region, and epidemiologic profile. According to the 2006 LCMS, 7.6% of the population (all ages) self-reports symptoms of fever/malaria in the 4 weeks before survey. Of this number, 90% either seek care at a facility or self-medicate. Given the lack of more specific data is it difficult to determine the number of fever cases that received Coartem within 24 hours of onset of symptoms – the 2006 MIS reports that 8% of children under 5 had done so. We choose the deliberately conservative supposition that by 2008 20% of fever cases had received Coartem within 24 hours of onset (here a conservative value is a higher value). A study of 4800 households comprising 240 communities (20 sampled households per community) in eight districts is sufficient to identify a increase of 20 percentage points in the proportion of the population either receiving Coartem or a diagnostic test within 24 hours of onset of fever in any of the treatment arms (public sector intervention, private sector intervention, or both), at a significance level of .01 and a power of .99. This is based on the following assumptions: • An average household size of 5 individuals. • Health seeking behavior at baseline as per the above discussion. • A hypothesized intra-cluster (within community) correlation of .10 with regards to individual health seeking behaviors. The same sample size and assumptions are also sufficient to discern a difference of 10 percentage points in the proportion of the population either receiving Coartem or a diagnostic test within 24 hours of onset of fever between any two treatment arms at a standard sifgnificance level of .05 and a power of .8. VII. OPERATIONAL/ MPLEMENTATION ISSUES 1. Government Support The evaluation is client driven. The GRZ initiated the discussion on piloting the overall ACT subsidy in the country. The project has been endorsed by both the Ministry of Health and the National Malaria Control Centre (NMCC). The government has expressed support for a rigorous impact evaluation and participated in the design, including representatives from the MOH, NMCC, and MSL. Partners, such as the DFID, Clinton Foundation, JSI, and the University of Zambia are also actively involved in the design and, to various degrees, in implementation. 2. Risks and Risk Mitigation Challenges include i) financial constraints to implement an ACT subsidy; ii) cross-border or other leakages of pharmaceuticals; iii) regulatory challenges related to the deregulation of ACT to make it available as over the counter (OTC) drug to enable the private sector ACT subsidy; iv) the determination of pricing, availability, and administration of RDTs in private sector outlets, sharps disposal, and development of treatment and referral protocols. 14 Mitigation measures to control for these potential risks include i) limiting the geographic coverage that assures operational feasibility and, at the same time, enables scientifically rigorous evaluation; ii) incentivizing wholesalers to use package tracing to enable better monitoring of leakages; iii) discussion with the National Regulatory Agency (NRA) assured that the use of ACTs in the private sector pilot is feasible if the study is limited to a few districts; iv) careful development of diagnosis, treatment, and referral protocols, which requires training of personnel, and high level discussions with the MOH, NMCC, and the NRA 3. Human Subject Review Ethical clearance will be obtained from the Ministry of Health. All work will be performed according to the international guidelines for human experimentation in clinical research specified in the World Medical Association Declaration of Helsinki (2000), built around the basic principal that “it is the duty of the physician in medical research to protect the life, health, privacy, and dignity of the human subject.” The proposed data collection includes tests for malaria and anemia, both of which require the drawing of a small amount of blood (one drop) from test subjects through finger-pricking. This process is painless and both tests use clean, sterile instruments that are completely safe. Neither test will be carried out before first obtaining the written consent of the subject or, in the case of minors, a parent or primary caretaker. Results are provided to the subject immediately following the test and are kept confidential. Given the presence of malaria or anemia in any test subject, a local doctor will be informed only with the prior written authorization by or on behalf of the subject. Proposed Implementation Time Line The implementation timeline depends on the pace of deregulation, and reaching a consensus on the optimal design of the interventions. The design and regulatory questions are coming to finalization. The MOH and NMCC have been encouraging the commencement of implementation. Given these, the baseline survey is planned to be fielded in December 2009. It is important that the interventions only start following the baseline to avoid contamination/bias in the findings. Commencement of the implementation of the public sector intervention is planned for around December 2009 or January 2010, The first follow up round is planned one year later, in December 2010. 15 Annex Figure 1: System A. Group A : District Stores plus Commodity Planner (CP) Medical Stores Limited : One pack for each Level 1, 2 and 3 Hospital and one for each district ly M PULL PULL th on on t hl M y CP receives stock from MSL and manages Hospitals receive stock district stock in from MSL District store room Districts Level 2 and 3 Hospitals CP processes and packs orders from HCs Level 1 Hospitals ly th M on on M t PUSH ce hl y wi T Receive facility packages Receive facility packages from CP from CP Health Centers with limited Health Centers with storage space adequate storage space Community Health Workers Annex Figure 2: System B. Group B : Cross Dock , No Commodity Planner Medical Stores Limited : One pack for each Hospital and Health Center ly M th on PULL PULL on th M ly District receives facility packages from MSL ; Hospitals receive stock No stock kept at from MSL District store Districts Level 2 and 3 Hospitals District repacks orders for HC s with limited storage District distributes facility packages to HCs Level 1 Hospitals y hl nt M Mo on th ice ly PUSH Tw Receive facility packages Receive facility packages from District from District Health Centers with limited Health Centers with storage space adequate storage space Community Health Workers Annex Figure 3: System C Group C : Cross Dock plus Commodity Planner (CP) Medical Stores Limited : One pack for each Hospital and Health Center y M t hl PULL PULL on on th ly M CP receives facility packages from MSL ; Hospitals receive stock No stock kept at from MSL District store Districts Level 2 and 3 Hospitals District repacks orders for HCs w ith CP distributes limited storage facility packages to HCs Level 1 Hospitals ly th Mo on M nt i ce hl PUSH y Tw Receive facility packages Receive facility packages from CP from CP Health Centers with limited Health Centers with storage capacity adequate storage capacity Community Health Workers . Annex Table 1. Public sector pilot districts, by type Province District CHW District? Group A - District stores plus commodity planner Luapula Mwense Luapula Milenge Yes Lusaka Luangwa Southern Choma Southern Kalomo Northwestern Kabompo Group B - Cross dock, no commodity planner Northern Nakonde Northern Mungwi Copperbelt Mpongwe Southern Monze Northwestern Mwinilunga Western Sesheke Yes Group C - Cross dock plus commodity planner Northern Kasama Yes Eastern Chama Central Mkushi Western Mongu Northwestern Chavuma Western Lukulu Comparison Group - No public sector intervention Eastern Nyimba Eastern Chadiza Lusaka Chongwe Yes Southern Siavonga Western Kaoma Southern Gwembe Annex Table 2. Private sector pilot districts and those covered in household survey Province District Public pilot type Private sector pilot? CHW district? Rural? Northern Kasama C Yes Yes No Luapula Milenge A Yes Yes Yes Luapula Mwense A No No No Eastern Chama C No No Yes Eastern Lundazi -- Yes No No Northern Chinsali -- Yes No Yes Eastern Chadiza -- No No No Eastern Nyimba -- No No Yes 19 ANNEX 2: Budget