Cameroon Performance-Based Financing Impact Evaluation Report Draft date: January 29, 2017 1 Table of Contents Executive Summary ...............................................................................................................4 Background ...........................................................................................................................6 Country context ...................................................................................................................................................................... 6 Health background ................................................................................................................................................................ 6 Health financing challenges in Cameroon.................................................................................................................... 7 Performance Based Financing .......................................................................................................................................... 8 PBF timeline in Cameroon .................................................................................................................................................. 8 Description of the PBF program design in Cameroon .......................................................................................... 11 Program overview ................................................................................................................................................................... 12 Service verification .................................................................................................................................................................. 12 Calculation of payments........................................................................................................................................................ 12 Research questions of the impact evaluation ........................................................................................................... 13 Methods ............................................................................................................................. 14 Treatment groups ................................................................................................................................................................ 14 Data sources ........................................................................................................................................................................... 16 Household surveys.................................................................................................................................................................... 17 Facility-based survey .............................................................................................................................................................. 18 Facility assessment module ................................................................................................................................................. 18 Health worker interview module ...................................................................................................................................... 18 Observations of patient-provider interaction module ............................................................................................ 18 Patient exit interviews ........................................................................................................................................................... 19 Statistical methods .............................................................................................................................................................. 20 Facility and catchment area exclusions......................................................................................................................... 20 Specifications ............................................................................................................................................................................. 20 Cameroon PBF Impact Evaluation Results ............................................................................ 22 Balance at baseline tables ................................................................................................................................................. 22 Comparison of operational funds available and subsidy payments to C1 and T1 at endline ............... 25 Analysis of household care seeking behavior........................................................................................................... 28 Household survey utilization results ........................................................................................................................... 32 Anthropometrics .................................................................................................................................................................. 36 Facility survey utilization results .................................................................................................................................. 36 Health care spending .......................................................................................................................................................... 41 Patient satisfaction .............................................................................................................................................................. 43 Satisfaction with antenatal care ....................................................................................................................................... 43 Satisfaction with child health consultations (< 5 years old) ................................................................................ 46 Health worker satisfaction and motivation............................................................................................................... 50 Health worker availability in the health facility ...................................................................................................... 56 Drugs and equipment in the health facility ............................................................................................................... 57 The quality of consultations for children under 5 years of age ........................................................................ 61 The quality of antenatal care ........................................................................................................................................... 62 Discussion ........................................................................................................................... 63 Health sector context .......................................................................................................................................................... 64 Impact evaluation design and research questions ................................................................................................. 64 MCH Coverage indicators ..................................................................................................................................................... 65 Out of pocket payments ......................................................................................................................................................... 65 Patient Satisfaction ................................................................................................................................................................. 65 Direct observation ................................................................................................................................................................... 65 Equipment and Staffing ........................................................................................................................................................ 66 Health worker motivation.................................................................................................................................................... 66 Comparison across intervention arms ........................................................................................................................ 66 Study limitations .................................................................................................................................................................. 66 2 Conclusions and Policy Recommendations........................................................................... 68 Appendix A Intervention group comparison table................................................................................................. 69 Appendix B Regional health facility maps ................................................................................................................. 70 Appendix C PBF subsidy table ........................................................................................................................................ 73 Appendix D PBF quality evaluation .............................................................................................................................. 76 Appendix E OLS health care shopping results.......................................................................................................... 92 Appendix F Results of household analysis with stratification on baseline facility bypassing ............. 93 Appendix G WHO well-being index............................................................................................................................... 94 References .......................................................................................................................... 95 3 Executive Summary  In order to more directly link payments and funding at the provider level with the quantity and quality of health services performed, Cameroon started a Performance Based Financing (PBF) pilot in the health sector in 2012 in 14 districts in three regions of the country: East, South West and North West. Performance-based financing is a health systems management tool designed to increase the efficiency of health system inputs to improve the coverage and quality of priority maternal and child health services by paying health facilities bonuses linked to the quantity and quality of services delivered.  The PBF pilot was accompanied by a prospective randomized impact evaluation. In order to distinguish the influence of the different components of the PBF reform, the evaluation compared four arms: (1) the standard PBF package, (2) the same level of financing but not linked to performance, and with the same levels of supervision, monitoring, and autonomy as PBF, (3) no additional resources or autonomy, but the same levels of supervision and monitoring as PBF, and (4) pure comparison. The randomization took place at the health facility level and overall the four study groups were well balanced at baseline.  The evaluation used a combination of household and health facility surveys conducted at baseline and endline to assess the impact of the interventions.  The research questions of the impact evaluation and the responses found are summarized in the table below. Table 1: Summary of research questions and findings Nr Research Question Effect Quantification/Details 1 Does the PBF program increase the Mixed Deliveries: No coverage of MCH services? ANC: No Vaccinations: Yes 2 Does the PBF program increase the Yes/Mixed Structural: Yes quality of MCH services delivered? Content of care: No 3 Is it enhancing monitoring & Clear effect for the importance of additional evaluation and supervision or the link financing plus reinforced supervision through between payments and results that PBF instruments (groups T1 and C1 vs groups leads to improvements observed in C2 and C3). quality or coverage? The difference in impacts between additional financing plus reinforced supervision through PBF instruments (C1) and full PBF (T1) is less clear. 4 What is the contribution of enhanced Very limited or supervision and monitoring to inexistent improving MCH service coverage and quality in the absence of increased autonomy or additional financial 4 Nr Research Question Effect Quantification/Details resources? 5 Does the PBF program lower informal Yes charges for health services? 6 Does the PBF program lower formal Yes, for laboratory and user charges x-ray fees. 7 Does the PBF program improve Yes, improvements in physical and social accessibility of equipment and patient health services? satisfaction. 8 Does the PBF program lower staff Yes, more qualified absenteeism? workers present. From a policy point of view, these impact evaluation results suggest the following take-away lessons: 1) In general, PBF is an efficient mechanism to bring payments and funding at the provider level, leading to significant increases in coverage (child and maternal immunization, family planning, HIV testing) and improvements in structural quality of care. It also leads to a decrease in out-of-pocket payments, in particular unofficial payments. 2) For many of those outcomes, the differences between the PBF group (T1) and the additional financing group (C1) are not significant. It should be noted that the C1 group offered all the elements of PBF except the direct link between individual facility performance and additional financing. It is not obvious these differences in intervention design have been salient enough for staff and management. 3) There was, however, a clear effect for the importance of additional financing plus reinforced supervision through PBF instruments (comparing groups T1 and C1 vs groups C2 and C3). Enhanced supervision and monitoring are not sufficient to improve MCH outcomes. 4) The absence of impacts for some MCH indicators such as skilled deliveries and ANC visits was surprising. It is possible that the supply-side incentives for providers were not sufficient given existing user fees which might act as a barrier on the demand-side. A policy discussion about combining demand-side and supply-side incentives would be useful. 5) In terms of quality of care, most of the positive impacts were observed on structural quality. However, despite an increase in providers and supplies available at health facilities, PBF did not increase the completeness of service provision (content of care) during antenatal care and child health consultations. Further reflection and efforts should be devoted to identify mechanisms to incentivize or otherwise improve the content of care beyond equipment, supplies and staff availability. 5 Background Country context Cameroon is a country of approximately 24.36 million people, with 54% of the population residing in urban areas. The country is classified as a lower-middle-income country with a GDP per capita of US$ 3,100 in 2015 (Central Intelligence Agency 2016). Relative to other countries in Central Africa, Cameroon has a favorable economic environment including modest oil resources which account for 40% of export earnings, productive agricultural conditions, and the availability of high value timber species, and minerals (Central Intelligence Agency 2016; The World Bank 2016a). In recent years, several large infrastructure projects have led to increased growth in public works and construction and services and the economy as a whole has experienced steady economic growth with annual GPD growth rates between 5.6 – 5.9% between 2013 and 2016 (Central Intelligence Agency 2016). Nonetheless, Cameroon suffers from relatively high levels of income inequality, weak governance, corruption and high levels of poverty (The World Bank 2016a). The poverty rate of 37.5% in 2014 represents a decline of less than 3 percentage points from the rate in 2001 (40.2%). The poverty rate is greater in rural areas where approximately 56.8% of the population lives in poverty, and this rate has increased over the last fifteen years from 52.1% in 2001. At 8.9% in 2014, the urban poverty rate is much lower and has declined from 17.9% in 2001 (Institut National de la Statistique 2015c). Life expectancy at birth is 58.5 years and has actually decreased slightly over the last decade (Bove et al. 2013). Sixty percent of the population in Cameroon is less than 25 years old and the fertility rate remains high at 4. . Health background Despite being one of the more wealthy countries in the Central African region, and the country’s relatively high health spending of $59 per capita in 2014 (The World Bank 2016b), Cameroon’s health indicators resemble countries that spend much less on health care (The World Bank 2016c). Cameroon did not achieve Millennium Development Goals 4 & 5 which called for large reductions in maternal and child mortality (MDGs) (United Nations Statistical Division 2015). Moreover, Cameroon is one of the few countries with high maternal mortality where maternal mortality did not decrease at all between 1990 and 2015 (Alkema et al. 2015). One in every 25 women of reproductive age in Cameroon continue to die from pregnancy-related causes (Institut National de la Statistique (INS) et ICF International 2012). Though many factors affect maternal mortality, one potential explanation for the country’s slow progress in achieving declines in maternal mortality is that the proportion of women delivering in health facilities has remained unchanged over much of the last decade. During the last ten years (between 2004 and 2014), the percentage of deliveries that were assisted by a skilled health professional increased from 61.7 percent to only 64.7 percent (ICF International 2012; Institut National de la Statistique 2015a). Skilled delivery assistance varies considerable by wealth and region in Cameroon. For instance, 21% of women in the poorest wealth quintile accessed skilled delivery care while almost 98% of 6 women in the highest wealth quintile gave birth with a skilled attendant. Skilled delivery is lowest in the Extreme North where approximately 29% of women deliver with a skilled provider compared to over 96% in West region (Institut National de la Statistique 2015b). Like skilled delivery, the percentage of women receiving four or more antenatal care visits, another health service with the potential to decrease maternal and child mortality, remained unchanged over the last decade. In 2004, 59.1% of women had received four antenatal visits, the number recommended by the World Health Organization, compared to 58.8% in 2014 (Institut National de la Statistique 2015b). Even when women obtain antenatal services, they still may not receive important service components due to low quality of care. Previous studies have shown that fewer than half of women are informed about the danger signs to look for during pregnancy, over 40% did not have the recommended number of tetanus vaccinations during their last pregnancy (ICF International 2012), and 25% did not provide a urine sample to test for protein, a required procedure to test for preeclampsia (The World Bank 2013a). Child mortality declined in Cameroon by approximately 21% between 1991 and 2014 (ICF International 2012; Institut National de la Statistique 2015a; Institut National de la Statistique (INS) et ICF International 2012); nonetheless, according to the most recent data approximately 1 in 10 children still die before their fifth birthday (Alkema et al. 2015; The United Nations Children's Fund 2015). The greatest reductions in child mortality were observed in the East (90 per 1,000 live births) and the South (50 per 1,000 live births), while the rate changed very little in Douala and Yaoundé. National averages obscure the large regional disparities that exist in child survival in Cameroon. Child mortality is over three times higher in North region (173 per 1,000 live births) than in the country’s two largest cities Doula (52 per 1,000 live births) and Yaoundé (42 per 1,000 live births) (Institut National de la Statistique 2015b). Health financing challenges in Cameroon Several aspects of the health care financing landscape in Cameroon contribute to the low quality of primary health care service provision, and sub-optimal coverage of essential maternal and child health care services. Cameroon spends $10 dollars more than the average for Sub-Saharan Africa (excluding South Africa) and has similarly high health spending to Senegal and Nigeria (Bove et al. 2013). However, despite this relatively high level of spending, the share of government spending on health is low and has not reached above 9 percent of the total budget in the previous 10 years (Bove et al. 2013). This amount falls short of the commitment of 15% of the annual budget to improve the health sector made by countries in the African Union in April 2001 in Abuja. Due to these low levels, in 2012 the share of per capita total health spending paid for by the government was only US $14 (i.e. 21.7 percent). Much of the remaining 70.4% of health spending is paid for through out-of-pocket users fees (Ministere de la Sante Publique 2016). Another important part of the problem is that the operational level receives a small fraction of the health budget. Although the health sector budget has more than doubled in recent years, the majority of these resources have been allocated for administration and infrastructure. This has resulted in a scarcity of funds to meet operating expenses incurred in the day-to-day business of a district health system (e.g., consumables, drugs, regular maintenance, community outreach, etc.) (Ministere de la Sante Publique 2016). Further, funds are not targeted towards the most underserved populations, or to health facilities operating in more challenging socioeconomic or security contexts (Ministere de la Sante Publique 2016). Along with high out- of-pocket costs for care and the low level of financing for primary health care, the unequal distribution of medical professionals in Cameroon may also contribute to inadequate coverage of essential health services. While there are 1.9 doctors per 10,000 people in Cameroon, most of 7 these health workers are employed in urban areas and over half work in just three regions: Center, Littoral and West (Bove et al. 2013). North region is home to 18% of the population but only 8% of physicians, and while 18% of the population lives in Center, this region contains 40% of the national physicians (Bove et al. 2013). Performance Based Financing Performance Based Financing (PBF) comprises a set health system reforms meant to increase the coverage and quality of essential health services, as well as efficiency and equity, often with a special focus on maternal and child health. PBF program models differ but all involve the purchasing of health services using a pre-defined list of services and prices (Fritsche, Soeters, and Meessen 2014). Performance-based financing also includes a strong verification system that relies on systematic and detailed review of health facility records as well as community-level client tracing whereby reported patients are asked a series of questions to confirm health care receipt (Fritsche et al. 2014). Quality is verified using quality checklists, which cover a variety of core domains. For instance, aspects covered under the water and sanitation domain might include the availability of an incinerator, latrines, and a water source, and the absence of organic waste or used syringes in the clinic. For antenatal care, privacy, and the availability of functioning equipment are assessed. In addition, the verification teams review a sample of patient records for antenatal care to assess adherence to recommended protocols. Many PBF programs also involve increasing health facility autonomy. The most essential components of autonomy as defined in the PBF Toolkit, include the choice to procure inputs locally rather than from a central supply, use of a bank account, autonomy to hire, fire and discipline staff etc. Finally, PBF also provides a number of tools and mechanisms to improve the management of the health facility. The first tool is the business plan, which outlines the baseline conditions at the health facility, sets goals in terms of expected results, and then describes strategies to achieve these goals (Fritsche et al. 2014). The second management tool is health care provider performance evaluation checklists, which assesses provider performance in areas such as timeliness, collaboration with colleagues, initiative, and the quality of their work as measured by their adherence to norms and standards. Finally, the third tool is the indice tool, which helps the facility to manage cash income and expenses and plan the facility budget. It includes revenues (e.g. user fees, PBF subsidies, and expenses (e.g. medicines, supplies & equipment) from the past quarter, quarterly financial activities, and the budget for performance bonuses (Fritsche et al. 2014). PBF timeline in Cameroon Cameroon’s first experience with PBF began with the Redynamisation des Soins de Sante à l’Est du Cameroun (REDSSEC) project. In 2006, REDSSEC implemented a pilot Performance Based Financing (PBF) program in Faith Based Organization (FBO) facilities in the East region with support from Cordaid and Catholic Relief Services. The project began with 4 FBO facilities in Batouri district, and then expanded to FBO facilities in Bertoua, Doume and Yokadouma districts through 2011 (Figure 1). In 2008, the World Bank approved a US$25 million loan to the government of Cameroon through the Bank’s Health Sector Support Investment Project (HSSIP). In 2011, through support from the HSSIP, a PBF pre-pilot began in the Littoral region covering four health districts. The program began in July 2012 in the North West and South West regions, with four districts 8 included in each region. In October 2012 the program expanded to the East region, covering all 14 health districts in the region. Out of the 26 health districts throughout Cameroon implementing PBF, 14 districts were included in the impact evaluation (see figure 2 and Appendix B). The other 12 (4 in Littoral and 8 in the East) had already begun implementing some form of PBF before the impact evaluation baseline survey was conducted (either through the Littoral pre-pilot or the REDSSEC project in the East), and were thus excluded from the evaluation. Figure 1: Timeline of PBF implementation in Cameroon 2006 2008 2011 July 2012 October 2012 • REDSSEC • WB • PBF pre-pilot • Program • PBF implemented approved $25 began in began in exapnded to PBF pilot in million to Littoral in North West East region East region in Cameroon four health and South covering all faith based for Health districts West regions, 14 districts organizations Sector with four in three Support districts districts Investment included in Project each region (HSSIP) 9 Figure 2: Cameroon PBF project and Impact Evaluation map 10 Description of the PBF program design in Cameroon 11 Program overview The administrative and technical aspects of the PBF program in Cameroon were managed at the regional level by Performance Purchasing Agencies (PPA). PPAs are autonomous entities that have a contractual relationship with the government of Cameroon who entrusts the agencies with the management of funding intended for health care providers. All PBF health facilities sign a Performance Contract issued by the PPA which described conditions required to obtain PBF subsidies. These requirements include the management improvement, minimum quality levels, governance and financial inclusion, and clauses for termination of the contract. Additionally, PBF facilities prepared quarterly business plans and used frameworks for the health administration linked to performance payments. All facilities were trained to use the indice tool; however, use of the indice tool varied among Cameroon PBF facilities. PBF contracts were signed for a period of 3 months. Health facilities with performance contracts were responsible for completing registers and a monthly activity report/declaration form. This report and registers were used to document reported health service provision and were used as the primary basis for service verification. A list of the health services subsidized by the program, and the subsidy amounts is located in Appendix C. A copy of this report was sent to the PPA each month. After the verification of the quantity of services provided and declared in the monthly report was completed, the bill of health facility was established and paid monthly. The declarations form verified and validated was used to justify the payment subsidies provided to the health facility. Service verification Health service verification was completed on a monthly basis by the Fund Holder Agency supervisors. The Fund Holder Agency supervisors used the facility register and tally sheets to verify that the number of services reported by the health facility in the payment request form was consistent with the facility documentation. Should the supervisor encounter any errors, these problems are corrected in the presence of the facility staff, and any fraudulent cases are tracked and documented. As an added means of quantity verification, a sample of patients for each health service subsidized was contacted either by phone or in person by local associations to confirm that they received the health service reported by the health facility and to assess patient satisfaction. Additionally, the District Medical Team in collaboration with the Fund Holder Agency assessed the quality of the health services provided by PBF health facilities. This assessment used a standardized checklist to verify that a minimum quality level is met, and to calculate a quality score for the health facility ranging from 0 – 100%. The quality score is used to calculate a quality bonus that is received by the health facility. The quality checklist is available in Appendix D. Calculation of payments The validated quantitative data, and the quality assessment were used to calculate performance payments for PBF health facilities. The quality bonus provided an increase of up to 30% of the total payment based on health service quantity. This percentage depended on the health facility quality score. Quality assessments were conducted quarterly, and focused on facility management, hygiene and sanitation, as well as specific attributes of service delivery. The services delivery items included, including among many other categories, listing user charges, privacy, the condition of the waiting area and consultation room, and the correct management of cases (full quality assessment in Appendix D). For example, if the health facility received a 65% quality score, and their total payment amount based on the services they provided was 597,240 12 CFA, the quality bonus provided to the health facility would be calculated as follows: 597,240 x 0.30 x 0.65 = 116,461 CFA, and the total payment to the health facility would be 597,240 + 116,461 = 713,701 CFA. Additionally, an equity bonus was included in the calculation of performance payments. The equity bonus was paid to health facilities that faced serious structural problems making service provision more challenging. Equity Bonuses ranged from 0% to 80% of the basic subsidy. Each region applied the criteria listed below differently. North West and South West applied the same scoring; however, few facilities in South West received the equity bonus since all almost all facilities in the region were located in urban areas. The East region had a slightly difference scoring approach but also used the same criteria. The following issues were considered in the calculation of this bonus:  Geographical inaccessibility (hard-to-reach) that makes it difficult the retain staff;  The size of the health area and low population densities that create viability issues (high running costs)  Extreme poverty The facility management committee had the authority to decide on the allocation of PBF revenue. These decisions must have been clearly documented in facility business plans. Research questions of the impact evaluation Over time, PBF has been implemented in a growing number of countries. Many studies have shown a positive association between PBF and health service coverage, and some with improvements in quality. An early impact evaluation in Rwanda where districts were randomly assigned to treatment (PBF) and comparison (input financing with matched financial resources) found large and statistically significant positive impacts on institutional deliveries and preventive care visits from young children and also on quality of prenatal care (Basinga et al. 2010). After this promising start, many other quasi-experimental studies that have shown similarly positive results , and several others have shown favorable results for many – though not all – outcomes assessed the research community has not reached a consensus about the effectiveness of PBF at increasing health service coverage (Huillery and Seban 2014). The PBF design in the Demogratric Republic of the Congo studied by Huillery and Seban 2014 deviated from the planned intervention design and what are now considered PBF best practices, potentially explaining the findings (The World Bank 2014). Moreover, few studies have examined the factors and mechanisms that influence the impact of PBF− an area of considerable operational significance since PBF often involves a package of constituent interventions: linking payment and results, independent verification of results, managerial autonomy to facilities and enhanced systematic supervision of facilities. As PBF had never been implemented in Cameroon on any meaningful scale and had never been systematically evaluated, our larger policy objectives for the impact evaluation are to (a) Identify the impact of PBF on maternal and child health (MCH) service coverage and quality, and to (b) Identify key factors responsible for this impact. In doing so, we expect that the results from the impact evaluation will be useful to designing national PBF policy in Cameroon and will also contribute to the larger body of knowledge on PBF. Though we are interested in a wide range of outcomes in this report, we consider the main outcomes in terms of coverage to be 13 ANC including anti-tetanus vaccination, skilled deliveries, vaccinations and family planning. We used skilled deliveries for the power calculations to determine the study sample size. The impact evaluation will focus on the following research questions: 1. Does the PBF program increase the coverage of MCH services? 2. Does the PBF program increase the quality of MCH services delivered? 3. Is it the enhanced monitoring & evaluation and supervision or the link between payments and results that leads to improvements observed in quality or coverage? 4. What is the contribution of enhanced supervision and monitoring to improving MCH service coverage and quality in the absence of increased autonomy or additional financial resources? In addition, the impact evaluation will also examine the following research questions that relate to intermediate outcomes in the hypothesized causal pathway: 1. Does the PBF program lower informal charges for health services? 2. Does the PBF program lower formal user charges? 3. Does the PBF program improve physical and social accessibility of health services? Accessibility of health services will be examined in terms of the convenience of facility opening hours, availability of services through outreach, client perceptions of convenience of accessing health services and client perceptions of health providers’ attitudes towards clients 4. Does the PBF program lower staff absenteeism? Methods This report and the methods described below are focused on the quantitative component of a mixed method impact evaluation. A qualitative midline study was conducted in 2014 and the endline study took place in 2016. Results from the qualitative component will be shared in a separate report in the spring of 2017. Treatment groups Table 2 below describes the 4 study groups formed by randomizing Medicalized Health Centers (CMAs), or primary health centers with a medical doctor on staff, and Integrated Health Centers (CSIs) (primary health care centers without a doctor). The randomization for this study was at the health facility level. From an operational and public health perspective, randomizing at the district level would have make more sense given the proximity of some facilities. Indeed, the risk with facility-level randomization is that neighboring facilities from different groups might learn from each other and apply principles outside their treatment group. However, this was not feasible given that the Government of Cameroon had already decided and announced which districts would be included in the PBF pilot. Randomization at the district level was therefore precluded. 14 Public randomization ceremonies were held in each region between February and June 2012, just prior to the launching of the PBF program in each region (The World Bank 2013b). All health facility management staff from health facilities in the districts covered by the evaluation attended the randomization ceremony. We hope to answer the main research questions identified by making comparisons between these groups. Table 2: Study groups T1: PBF with health worker performance C1: Same per capita financial resources as bonuses PBF but not linked to performance; Same supervision and monitoring and managerial autonomy as T1 C2: No additional resources but same C3: Status quo supervision and monitoring as PBF arms and T1 and C1 *See Appendix A for detailed description For the purposes of our study, the ‘full’ PBF package of interventions included the following elements:  Linking payment and results, including performance bonuses for health workers  Independent monitoring of results  Systematic supervision of health facilities defined as regular supervision by an external supervisor from the district hospital team using a structured checklist and providing immediate feedback to facility staff on problems identified and potential solutions to improve service delivery. Systematic supervision included monitoring whether the facility is complying with national user fee guidelines  Limited managerial autonomy to facilities defined as autonomy over use of resources combined the ability to hire additional staff using health facility income and managerial discretion Facilities in group T1 implemented this full PBF package. Facilities assigned to group C1 received a fixed per capita budgetary supplement that matches the per capita budgetary allocation for T1 facilities. However, this supplement was not linked to performance. C1 facilities received the same supervision and monitoring and managerial autonomy over the budgetary supplement received. Both T1 and C1 facility managers had the autonomy to hire staff with their PBF revenues or budgetary supplement received, and also to fire these staff if necessary. T1 and C1 facility managers also had the autonomy over how to use these revenues. C2 facilities received no additional resources but the same supervision and monitoring as T1 and C1 facilities. District-level supervisors responsible for supervising T1, C1 and C2 facilities used the same tools and received the same supplementary payments for visits to facilities in these three groups. However, quality scores were linked to facility payments only in the case of T1 facilities. C3 facilities were the ‘business as usual’ facilities and did not receive any additional resources or inputs. C2 and C3 facility managers did not have the autonomy to hire/ fire staff or financial autonomy. National user fee caps, and facility user fee rates, were published on a signboard placed in all study group health facilities. The IE team also included monitoring of adherence to national guidelines as part of the monitoring and supervision intervention in T1, C1 and C2 facilities. As the status quo group, the C3 facilities did not receive this additional monitoring & supervision. The number, type, and percent private of study health facilities in each study district are shown in Table 3. All public and private health facilities in the 14 study districts that were officially 15 registered with the Ministry of Public Health were eligible for inclusion in the study. All district hospitals in 14 health districts were included in the full PBF (i.e., treatment) arm. This is because district hospitals play a critical role in supervising and acting as source of referral services for all facilities in the district. District hospitals did supervise and support treatment and comparison group CMAs and CSIs differently based on the group they are assigned to. Household and facility-based surveys were be implemented in district hospitals and households associated with their catchment areas1 in the 14 pilot districts to gain insights into the role that district hospitals are playing in the 4 study groups. However, these data will not be used for making inferences about the impact of PBF, and are not included in the analyses presented in this report. Table 3: Summary of health facilities included in the Impact Evaluation District Number of health facilities CSI CMA District Confessional For- Total Private Public Public Hospital CSI/ CMA/ profit/ (%) Hospital Para- public Abong-Mbang 14 2 1 4 2 23 27% Doume 9 1 1 2 1 14 23% Lomie 7 2 1 2 0 12 18% Messamena 9 1 1 2 0 13 17% Nguelemendouka 5 0 1 1 0 7 17% Kette 9 0 1 0 0 10 0% Total in East 53 6 6 11 3 79 19% Kumbo East 17 2 1 6 4 30 34% Nkambe 11 2 1 4 2 20 32% Ndop 12 2 1 8 4 27 46% Fundong 9 3 1 12 3 28 56% Total in North West 49 9 4 30 13 105 43% Mamfe 11 1 1 1 0 14 8% Kumba 10 1 1 5 1 18 35% Buea 10 3 1 0 9 23 41% Limbe 10 1 1 1 7 20 42% Total in South West 41 6 4 7 17 75 34% Pilot Zone total 143 21 14 48 33 259 33% Data sources The evaluation relied on two main sources of data to answer the impact evaluation research questions identified: 1Some villages will not fall within the catchment areas of other CSIs and CMAs in the district. Households in these villages will be excluded from the sample for the impact evaluation. 16 1. Household surveys: A household survey implemented at baseline (i.e., before implementation of PBF began), and at endline (i.e., after PBF was implemented for two years). 2. Facility-based surveys: A facility-based survey was also implemented at baseline and at endline. Household surveys The impact evaluation adapted the Health Results Innovation Trust Fund (HRITF) survey instruments for this impact evaluation. Household surveys were conducted in each of the 14 districts included in the impact evaluation. To select the households to be surveyed, a catchment area was first established for each of the 245 primary care facilities. GIS mapping was conducted before the baseline survey to define realistic catchment areas for health facilities. GIS mapping was necessary because the government does not have a clearly defined health map with specific catchment areas. The government defines instead “Health Areas� (similar to sub- districts) that often include several facilities. As such it was necessary to define “zones of responsibility� for each facility. This GIS mapping defined ‘true’ catchment areas by taking into account physical features (like terrain or water bodies) and roads that influence travel time and thereby potentially affect health facility choice. One village from each health facility catchment area was randomly selected for the household survey. Regional maps of the study health facilities are presented in Appendix B. A village household listing exercise was first conducted to identify all village households. At baseline 16 of the households identified in the listing exercise were randomly selected to be surveyed in each village. The survey team attempted to revisit all baseline households at endline. However, many baseline households could not be located or were no longer eligible at endline. When this occurred, baseline households were replaced using the nearest neighbor as recorded in the listing exercise. An additional four households were added to the household sampling roster at endline such that a total of 20 households were sampled in each village for the endline survey. In both rounds, the primary inclusion criteria for the household survey was that household must have contained at least one woman who had been pregnant in the 24 months preceding the survey. Though the sample was meant to be a panel, with repeat sampling of the same households at baseline and endline, only a small proportion of households sampled at endline were also sampled at baseline. For instance, only 29% of the 4,813 households from which the 6,275 pregnant women surveyed at endline resided were also sampled at baseline. Therefore, the surveys are analyzed as repeated cross-sectional surveys rather than panel data in this report. The household survey was administered to all members of the household who were present on the day of data collection. Demographic data including educational attainment and labor force participation was collected from all adult members of the household. Data on recent illness and health care use in the past four weeks was collected from all household members, with primary care givers providing information about child health. Household level data on housing characteristics, household assets and household level income was provided by the head of household. Additionally the household survey contained separate modules for women of reproductive age (15 – 49 years), women who had been pregnant in the 24 months before the survey, and for children under five years of age. The main health themes covered in these modules included:  Health behaviors for MCH services  Health seeking behaviors, barriers to use and health service use 17  Household health expenditures  General perceptions of health service quality In addition, the survey teams weighed and measured the height of all children aged under 5 years present in the household during the survey team’s visit. Facility-based survey The facility survey was conducted in all the CMAs, CSIs and District Hospitals in the 14 districts included in the impact evaluation. All facility team visits were unannounced. The facility-based survey included multiple components. The sample of health workers, patient- provider observations and client exit interviews was selected to enable findings from these three components to be linked. Facility assessment module The facility assessment module collected data on key aspects of facility functioning and structural aspects of quality of care. The individual in charge of the health facility at the time when the survey team visited the health facility was asked to be the respondent for this survey module. The main themes that were covered by the facility assessment included:  Facility staffing, including the staffing complement of the facility, staff on duty at the time of the survey team’s visit and staff present at the time of the survey team’s visit  Facility infrastructure and equipment  Availability of drugs, consumables and supplies at the health facility  Supervision  Record keeping and reporting to the Health Management Information System  Facility management  Official user charges at the facility  Revenues obtained at the health facility, and how revenues have been used Health worker interview module For health facilities with more than five health workers, a list of all clinical staff who worked in the area of maternal and child health providing prenatal or under five consultations was obtained. If this list contained more than five people, study enumerators interviewed a random sample of these health workers. If the list contained fewer than five people, all clinical personnel working in maternal and child health were interviewed. The interviews focused on the following areas:  Role and responsibilities of the interviewed health worker  Compensation, including delays in salary payments  Staff satisfaction and motivation Observations of patient-provider interaction module While the health worker interview module collects information on what health workers know, the purpose of this module is to gather information on what health workers actually do with their patients. A member of the survey team observed consultations with a systematic random sample of patients under five presenting with a new condition (i.e., not for follow-up visits or routine) and 18 new ANC clients. The observer used a structured format to note whether key desired actions were carried out. In the case of patients under five, the instruments were focused on whether IMCI protocols are followed. For ANC clients the instruments examined whether key desired actions (including counseling) were carried out. As primary care facilities do not offer ANC services on all days of the week – typically these are offered 2 days each week – the ANC module was not conducted at all health facilities. During the baseline survey, 5 under-5 and 5 ANC observations were conducted at each facility where these modules are implemented. After finding that many health facilities did not offer ANC on the day of the survey at baseline, during the endline survey enumerators were asked to interview as many women receiving ANC on the day of the survey as possible to increase the sample size. All health workers selected for patient- provider observations will be included in the health worker interview sample. Patient exit interviews Enumerators conducted an exit interview with all patients whose consultation was observed as part of the study procedures. If the patient was a child, the child’s caregiver was interviewed. The under-fives included in the patient exit sample were the same children whose consultation with a provider was observed. In addition to this, exit interviews were conducted with all ANC clients whose consultation with a provider was observed. Table 4: Data sources for the Impact Evaluation Data Who Level Frequency Description of Data Household Currently pregnant Household Twice: Health service use, health survey women; Women Baseline & care seeking behaviors who have had a endline and barriers to use for child in the 2 years MCH services, health preceding the expenditures, perceptions survey of health service quality Household Currently pregnant Household Twice: Height and weight survey women, non- survey Baseline & measurements pregnant women Endline who have had a child in the 2 years preceding the survey, children under five Facility Facility in-charge Facility Twice: Facility staffing, assessment Baseline & infrastructure, drugs Endline supply, equipment, supervision, HMIS reporting and management, user charges, facility revenue Health worker Health care workers Facility Twice: Staff work load, interviews Baseline & compensation, Endline motivation, and satisfaction Patient- First time ANC Facility Twice: Treatment and counseling provider clients Baseline & provided to patients. observation New under-5 Endline (Under-five & patients for curative ANC) care 19 Patient exit First time ANC Facility Twice: Patient’s (or caretaker’s) interviews clients Baseline & perception of quality of New under-5 Endline care and satisfaction patients for curative care New over-5 patients for curative care Statistical methods Facility and catchment area exclusions Specifications displayed below: 𝑌𝑖𝑗𝑡 = 𝛼𝑗 + 𝛾2015 + 𝛽1 𝑇1𝑗 𝐼2015 + 𝛽2 𝐶1𝑗 𝐼2015 + 𝛽3 𝐶2𝐼2015 + 𝛽 ′ 𝑋𝑖𝑡 + 𝜖𝑖𝑗𝑡 Where 𝑌𝑖𝑗𝑡 is receipt of the health service for woman/pregnancy i in enumeration area j in survey year t. 𝛼𝑗 is an enumeration area fixed effect, 𝛾2015 is a dummy variable that is equal to 0 in 2012 (baseline) and 1 in 2015 (endline). T1, C1, and C2 are dummy variables that are equal to 1 when the enumeration area was assigned to the each treatment group respectively and zero otherwise. The treatment variable is based on the assigned catchment area where the household is located; however, this may not have been the health facility were the household sought health care. 𝛽1 𝑇1𝑗 𝐼2015 , 𝛽2 𝐶1𝑗 𝐼2015 , 𝑎𝑛𝑑 𝛽3 𝐶2𝐼2015 are interaction terms between each the T1, C1, and C2 groups and the post indicator. These interaction terms measure the treatment effect in each group and can be interpreted as the difference in the change in health service use over the study period between the control group, and each treatment group respectively. 𝛽 ′ 𝑋𝑖𝑡 is a vector of control variables at the individual level (age, marital status, education level, religion, ethnicity, working status and type of work), and at the household level (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Control variables were included in all analysis of household level data. : 𝑌𝑖𝑡 = 𝛼𝑖 + 𝛽1 𝐼2015 + 𝛽2 𝑇1𝑖 𝐼2015 + 𝛽3 𝐶1𝑖 𝐼2015 + 𝛽4 𝐶2𝑖 𝐼2015 + 𝛽 ′ 𝑋𝑖 + 𝛽 ′ 𝑋𝑖𝑡 + 𝜀𝑖𝑡 A similar specification was used in most facility level analysis; however, in all analyses involving direct observation or exit interview data we have used an alternative specification 20 without facility fixed effects. The alternative specification was chosen after finding that the variable measuring the duration of ANC visits contained many outliers, and the use of fixed effects produced results that diverged widely from changes observed descriptively by comparing means. Therefore this analysis instead included treatment group dummy variables to control for baseline differences between groups. For consistency, all analysis of women sampled from antenatal care, and of caregivers sampled from child health consultations used this alterative specification. Additionally, sampling for these services was limited in many health facilities because antenatal care is only provided on certain days or the week, and due to low patient flows in smaller facilities. For this reason, analysis of patients sampled from health facilities was not restricted to health facilities represented in both the baseline and endline survey data. Analysis at the facility level included the following time invariant controls: type of health facility (public/religious/private) and location of the health facility (urban/rural). Additionally, when the analysis was at the individual level (i.e. women sampled from ANC visits, care givers sampled from child health consultations) the following individual level controls were also included: age, sex, marital status, and education level. 21 Cameroon PBF Impact Evaluation Results Tables 5 & 6 below display the baseline levels of individual and household level characteristics from the household survey. Group level means are compared individually using two-sample statistical testing, and F-tests were conducted to test for overall differences in the four study groups. The tables show that there were several statistically significant differences between study groups at baseline for individual level variables. In particular, the study groups were not balanced at baseline on religion and ethnicity. However, the groups appear balanced on many other dimensions including educational attainment, employment. The groups were also balanced for most categories of marital status though a greater proportion of women sampled in the T1 group described their marital status as in union. The study treatment groups were generally well balanced on household level characteristics including household composition, type of household and household ownership. However, the study groups were not balanced at baseline on the type of water source used at the household, and on household sanitation type. Balance at baseline tables Table 5: Individual level characteristics of household members sampled at baseline Individual all Mean Mean Mean Mean Mean p-value p-value p-value F- household members T1 C1 C2 C3 total T1/C3 C1/C3 C2/C3 statistic N Age 18.41 18.76 18.00 18.35 18.37 0.863 0.266 0.324 0.210 19232 Catholic 0.44 0.37 0.32 0.36 0.37 < 0.001 0.331 < 0.001 < 0.001 19196 Protestant 0.36 0.40 0.42 0.43 0.40 < 0.001 0.001 0.081 < 0.001 19196 Other religion 0.15 0.15 0.13 0.14 0.14 0.038 0.121 0.593 0.024 19196 Muslim 0.05 0.08 0.13 0.07 0.09 < 0.001 0.024 < 0.001 < 0.001 19196 Kom 0.08 0.08 0.05 0.14 0.09 < 0.001 < 0.001 < 0.001 < 0.001 19178 Banso 0.06 0.12 0.09 0.05 0.08 0.099 < 0.001 < 0.001 < 0.001 19178 Other ethnicity 0.86 0.80 0.86 0.81 0.83 < 0.001 0.591 < 0.001 < 0.001 19178 Adults > 18 years Years of school 5.65 5.70 5.57 5.52 5.61 0.214 0.084 0.615 0.315 6807 Literacy 0.74 0.75 0.72 0.73 0.73 0.498 0.162 0.627 0.251 7991 Any school 0.88 0.87 0.87 0.85 0.87 0.031 0.329 0.299 0.197 7984 Work 0.74 0.72 0.73 0.71 0.73 0.055 0.898 0.200 0.162 7812 Agricultural work 0.60 0.57 0.58 0.58 0.58 0.337 0.626 0.881 0.494 5698 Work in retail 0.14 0.16 0.16 0.15 0.15 0.370 0.833 0.827 0.631 5698 Other type of work 0.19 0.19 0.19 0.19 0.19 0.768 0.723 0.730 0.983 7737 Never married 0.17 0.18 0.18 0.18 0.18 0.268 0.931 0.745 0.677 8038 Monogamous marriage 0.45 0.46 0.47 0.45 0.46 0.781 0.645 0.321 0.593 8038 Polygamous marriage 0.07 0.08 0.08 0.07 0.07 0.471 0.582 0.423 0.431 8038 In union 0.21 0.17 0.17 0.19 0.18 0.079 0.117 0.224 0.003 8038 Divorced or widowed 0.11 0.11 0.10 0.10 0.10 0.791 0.409 0.572 0.565 8038 22 Table 6: Household level characteristics of households sampled at baseline 23 Tables 7 & 8 display facility characteristics and health service coverage at baseline. Unlike balance at the household level, the facility sample appears well balanced at baseline on all characteristics assessed. The sample was well balanced for most services assessed; however, we found statistical differences at baseline for ANC, self-reported childhood vaccination coverage, and bednet use among children under 5 years of age. Table 7: Facility level characteristics at baseline Mean Mean Mean Mean Mean p-value p-value p-value F- Facility T1 C1 C2 C3 total T1/C3 C1/C3 C2/C3 statistic N Number of beds in the health facility 8.07 9.98 11.70 9.26 9.84 0.447 0.801 0.252 0.444 185 Electricity in the health facility 0.70 0.78 0.69 0.77 0.73 0.410 0.913 0.363 0.632 206 Piped water in the health facility 0.40 0.38 0.35 0.35 0.37 0.663 0.791 0.926 0.947 206 Facility has an incinerator 0.08 0.22 0.24 0.23 0.19 0.030 0.913 0.931 0.109 206 Latrine in the health facility 0.85 0.84 0.85 0.79 0.83 0.452 0.537 0.402 0.831 206 Facility open 24 hours 0.66 0.72 0.64 0.71 0.68 0.605 0.898 0.438 0.776 206 Water towel and soap in Examination Room 0.46 0.43 0.47 0.45 0.45 0.896 0.857 0.803 0.977 199 Secure Box for Sharps 0.80 0.86 0.80 0.83 0.82 0.706 0.712 0.667 0.845 200 User Fees for Consultation Posted 0.38 0.32 0.36 0.35 0.35 0.809 0.721 0.920 0.941 206 User Fees for Laboratory Services Posted 0.34 0.35 0.37 0.23 0.32 0.254 0.224 0.156 0.514 195 Child Weighing Scale 0.87 0.88 0.94 0.83 0.88 0.590 0.455 0.059 0.322 202 Height Measure 0.41 0.43 0.45 0.53 0.46 0.250 0.301 0.427 0.663 191 Tape Measure 0.96 0.98 1.00 0.96 0.98 0.902 0.533 0.122 0.491 204 Blood Pressure Instrument 0.86 0.90 0.87 0.85 0.87 0.835 0.462 0.775 0.907 199 Thermometer 0.98 0.94 0.95 1.00 0.97 0.334 0.082 0.101 0.286 204 Stethoscope 0.96 0.92 0.91 0.91 0.93 0.344 0.951 0.918 0.745 202 Lab services 0.74 0.80 0.82 0.77 0.78 0.684 0.725 0.552 0.754 206 Blood test 0.34 0.42 0.48 0.54 0.45 0.083 0.311 0.570 0.361 159 Malaria test 0.97 1.00 0.91 0.97 0.96 0.970 0.301 0.244 0.168 160 TB test 0.13 0.28 0.20 0.19 0.20 0.496 0.374 0.863 0.475 159 HIV test 0.11 0.23 0.18 0.22 0.18 0.190 0.879 0.699 0.500 158 Facility provided immunization 0.98 0.96 0.95 0.98 0.97 0.944 0.582 0.377 0.713 206 Facility provides ANC 0.98 0.98 0.98 1.00 0.99 0.339 0.325 0.348 0.821 206 24 Table 8: Baseline health service coverage Comparison of operational funds available and subsidy payments to C1 and T1 at endline One important feature of the impact evaluation design was that the subsidies provided to control group 1 (C1) – which were not linked to performance – should be equal to the payment amounts provided to facilities in the full PBF treatment group. At endline, the health facility survey collected data on the amount of funding available at the health facilities for operation expenses, including the amount of revenue collected from cost recovery, funding provided directly by the Ministry of Health, and subsidies from the PBF program. To verify the equal receipt of financing, we compared data on total financing available, and subsidy amount received between T1 and C1 health facilities. The first panel of the table below displays the average amount of health facility revenue from all of these sources during each trimester of 2014 in T1, the full PBF group, and C1, the increased financing group. The second panel compares the amount of subsidies received in each of the two groups. Neither the subsidy amount, nor the total amount of financing available at the health facility differed statistically between the two groups during any trimester of 2014. We also sought to assess whether per capita payments were equal between groups, as this was the intention of the study. However, exact catchment area population numbers are not known. Therefore, we standardized payments by the number of health workers in each facility as a proxy measure for catchment area population. Using this proxy measure, we found no differences in payment amounts between groups (Table 9, panels 3 & 4). We also confirmed the subsidy data collected in the facility questionnaire by requesting subsidy data from the regional funds. This data, which represents the payments made by the regional funds to the health facilities in treatment groups T1 (full PBF) and C1 (additional financing), is presented in figures 3 – 5 below. These figures also show that the total payments provided to the health facilities in each treatment group were equal during the entire study period. 25 Table 9: Operational financing available at the health facility and PBF payments assessed at endline Total (cost recovery, Ministry of health, subsidies) T1 (full PBF) C1 (increased financing) p-value Q1-2014 received 3420226 4163127 0.5281 Q2-2014 received 3339472 4044135 0.5143 Q3-2014 received 3799585 4395996 0.748 Q4-2014 received 3876873 3873040 0.9979 n 53 48 101 Subsidies T1 C1 p-value Q1-2014 received 1322834 1725858 0.2009 Q2-2014 received 1241536 1751481 0.1805 Q3-2014 received 1757934 1401405 0.5654 Q4-2014 received 1428642 1264029 0.7897 n 53 48 101 Standardized by number of health workers T1 (full PBF) C1 (increased financing) p-value Q1-2014 received 587480 559007 0.7962 Q2-2014 received 442595 550209 0.2212 Q3-2014 received 436623 471578 0.6895 Q4-2014 received 467895 528527 0.5869 n 49 43 Subsidies Q1-2014 received 295233 391902 0.2906 Q2-2014 received 242738 361116 0.1223 Q3-2014 received 238236 260399 0.7039 Q4-2014 received 212285 312874 0.2159 n 49 43 Figure 3: Total payment provided to T1 and C1 health facilities in North West 250,000,000 200,000,000 150,000,000 CFA 100,000,000 50,000,000 0 July - 2013 2014 January - June December 2012 Total T1 Total C1 2015 26 Figure 4: Total payments provided to T1 and C1 health facilities in South-West 200,000,000 160,000,000 120,000,000 CFA 80,000,000 40,000,000 0 July - 2013 2014 January - June December 2012 Total T1 Total C1 2015 Figure 5: Total payments provided to T1 and C1 health facilities in East 70,000,000 60,000,000 50,000,000 40,000,000 CFA 30,000,000 20,000,000 10,000,000 0 July-December 2013 2014 January-June 2015 2012 Total T1 Total C1 While the payment results show that overall the two groups received equivalent financing volumes for the same number of health facilities (approximately 50 each) during the study period, when looking at per capita financing we find that the C1 group in fact received higher levels of per capita financing than the T1 group. This is largely due to the fact that in the South- West region, while the T1 and C1 groups had the same number of facilities (21 for T1 and 20 for C1), the total population covered by these health facilities varied substantially. In the T1 group several health facilities had very large catchment areas, resulting in a target population approximately three times higher than in the C1 group. As such the per capita payments in the South-West region were three times higher for the C1 group than the T1 group, which also affects the overall annual per capita payments (Figure 6). That being said, the overall payments for each group, when combining the three regions, shows equal payments across the two groups (Figure 7). Populations of catchment areas in Cameroon should also be interpreted with precaution given the lack of an up-to-date and comprehensive national health map. 27 Figure 6: Per capita subsidy payments, T1 and C1 facilities, $US 2.50 2.00 1.50 USD 1.00 0.50 0.00 July-Dec 2012 2013 2014 Jan-June 2015 Per capita T1 Per capita C1 Figure 7: Total subsidy payments, T1 and C1 facilities, $US 800,000 700,000 600,000 500,000 USD 400,000 300,000 200,000 100,000 0 July-Dec 2012 2013 2014 Jan-June 2015 Total T1 Total C1 Analysis of household care seeking behavior For the household survey, a random sample of 16 to 20 households was selected in each health facility catchment area. The analysis of the household survey in this report starts from the assumption that household members seek care in the health facility closest to where they live, or in other words that people living in the catchment area of a facility obtain health care in that facility. However it is apparent from the baseline survey data that households do not always seek care from the closest health facility in their health zone (table 10). The household survey analysis assigns a treatment group (PBF, C1, C2 or C3), to each household in the study, which represents the treatment assignment of the closest health facility to the sampled household. Additionally, both the baseline and the endline household survey included information about the name of the health facility where the household sought reproductive health care. Using this information, we 28 created a variable that measures whether women sought care in a health facility consistent with their assigned treatment group, a health facility assigned to another treatment group, and non- randomized hospital, or a health facility outside of the study area. At baseline, for antenatal care, for example, 44.8% of women sought care in a health facility assigned to their own treatment group, but 11.1% sought care in a higher level hospital (not included in the randomization conducted for the impact evaluation) and 22.2% sought care in other health facilities beyond their own health zone (18.7% in other facilities assigned to other treatment groups in our study sample and 3.2% in facilities not included in the study sample). Another 7.8% did not seek any antenatal care and we are missing information about the service location for the remaining 15.7%. If we focus on women for whom we have information about the service location, 52.3% sought care in in their “assigned� treatment group, and if we further exclude women who did not seek any antenatal care, this percentage increases to 57.6%. This “health care shopping� behavior whereby households bypass the closest health facility is also present for deliveries and postnatal care: at baseline, focusing on women for whom we have information about the service location and who sought care in a facility, only 51.9% delivered, and only 56.1% sought postnatal care in a facility with their corresponding treatment group. At endline, those percentages are slightly higher, but not very substantially: focusing again on women for whom we have information about the service location and who sought care in a facility, 60.9% obtained antenatal care, 55% delivered, and 60.6% sought postnatal care in a facility assigned to their treatment group. Table 10: Health care seeking behavior† Delivery Postnatal Baseline Antenatal care care care N % N % N % Did not receive the health service 214 7.78 585 21.26 1,825 66.34 Received the health service in assigned treatment group 1,231 44.75 951 34.57 440 15.99 Received the health service in different treatment group 514 18.68 361 13.12 154 5.6 Received the health service in a facility outside the study area 88 3.2 72 2.62 44 1.6 Received the health service in a non-randomized hospital 304 11.05 304 11.05 138 5.02 Missing data on service location 398 15.73 332 19.03 141 15.38 N 2,751 2,751 2,751 Delivery Postnatal Endline Antenatal care care care N % N % N % Did not receive the health service 226 6.82 582 17.57 1,954 59 Received the health service in assigned treatment group 1,626 49.09 1,358 41 705 21.29 Received the health service in different treatment group 435 13.13 395 11.93 177 5.34 Received the health service in a facility outside the study area 315 9.51 200 6.04 102 3.08 Received the health service in a non-randomized hospital 292 8.82 321 9.69 171 5.16 Missing data on service location 414 13.43 261 12.58 194 14.38 N 3,312 3,312 3,312 †Percentages calculated from household survey data among sampled women who had been pregnant in the 24 months before the survey. 29 When the household is indeed seeking care in a health facility that is consistent with their assigned treatment group, this assignment is correctly done. However, when the household seeks care in another facility, this assignment between household and study group is potentially erroneous, leading to measurement error. This measurement error would introduce statistical noise in the analysis and reduce our capacity to measure potential impacts of the interventions (attenuation bias). Another interpretation of these patterns is to see it as non-compliance with assigned treatments. The ITT model estimated remains valid. The causal estimates the ITT model creates may however not fully capture the causal effect of the treatment relative to a “clean� control, but rather measure the causal impact of having a treated facility closer compared to people living further away from a treated facility. These estimates are likely below the true causal effect of the intervention. This is a substantial limitation of the household survey analysis that needs to be kept in mind. The statistical analysis and interpretation of the household survey would be further complicated if this health care “shopping� behavior was driven or reinforced by the introduction of PBF or the interventions implemented in C1 (additional financing) and C2 (enhanced supervision). If this was the case, this could introduce a bias in the estimates going further than the attenuation bias described above. Tables 11 to 13 investigate whether the implementation of PBF or the other interventions have directly influenced household’s health care seeking behaviors. For antenatal care, deliveries and postnatal care, they report results from multinomial logit difference-in-differences regression models where the four options for the household are: not seeking care, seeking care at the assigned facility, i.e. the closest one in the health zone, seeking care in an unassigned facility, i.e. another facility of the same level potentially randomized into a different impact evaluation group, and seeking care at a non-randomized facility, generally a higher level hospital which were not included in the randomization. Overall, the results do not suggest that the health care seeking behavior is driven or even significantly influenced by the introduction of PBF or the other interventions in C1 and C2. We also conducted this analysis using OLS with a binary outcome equal to 1 if the respondent sought care in a health facility in their assigned treatment group, and zero otherwise. We found no evidence that health care shopping was affected by PBF in this analysis, consistent with the results from the multinomial logit (Appendix E). Health care shopping behavior by households was widespread in Cameroon at baseline in 2012 and continues to be widespread at endline in 2015, but does not appear to be a consequence of the introduction of PBF. We therefore decided to keep the presentation of the household survey results in which the analysis assumes that household members seek care in the health facility closest to where they live. We recognize that this assumption is not always verified and that therefore the results might suffer from attenuation bias. In addition to the fact that overall we did not find significant evidence that health care seeking behavior was influenced by the introduction of PBF, the following other considerations motivated our choice: 1) In many cases, the data collected about which health facility was visited allowed us to find out whether the visited facility was the closest one, but in case it was not, did not allow us to ascertain to which study group the visited facility pertained. 2) When we could ascertain to which study group the visited facility pertained, such bypassing behavior is clearly endogenous and assigning to the household bypassing its closest facility the study group of the facility actually visited would lead to endogeneity bias. 30 3) The results from the household survey analysis are broadly consistent with the results from the health facility survey analysis, which are not affected by the measurement error introduced by the health care shopping behavior of households. 4) The health care shopping behavior prevalent in Cameroon is likely present in many other countries. Our analysis uses the same assumptions and methods as the other impact evaluation reports including household survey results. However, to our knowledge, the household survey analysis in Cameroon is, so far, the only one to have explicitly collected or used detailed information about the name of the facility visited by the household sought care. We are therefore in a position to better acknowledge this study limitation and document how our household survey results are potentially affected by this health care bypassing behavior. Table 11: Health care shopping for antenatal care† ANC in assigned ANC in unassigned treatment group treatment group ANC in non- No ANC facility facility randomized facility p- p- p- p- b se value b se value b se value b se value Post indicator 0.001 0.014 0.040 0.045 -0.060* 0.033 -0.014 0.018 0.001 0.014 0.040 0.045 PBF/Post interact 0.012 0.018 0.065 0.055 -0.021 0.039 -0.019 0.024 0.012 0.018 0.065 0.055 Control 1/Post interact -0.004 0.018 0.016 0.054 0.045 0.038 -0.015 0.023 -0.004 0.018 0.016 0.054 Control 2/Post interact 0.015 0.020 -0.017 0.057 0.005 0.041 0.016 0.023 0.015 0.020 -0.017 0.057 N 5354 5354 5354 5354 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from multinomial logistic difference-in-differences regression models examining the effect of PBF on facility bypassing for reproductive health care. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work), household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation) and facility-level control variables at baseline (availability of electricity, piped water, and a latrine respectively at the health facility, facility open 24-hour a day, type of health facility, urban/rural and number of health workers employed at the health facility). Standard errors were clustered at the health facility level. Table 12: Health care shopping for skilled delivery† Skilled delivery in Skilled delivery in Skilled delivery in assigned treatment unassigned treatment non-randomized No skilled delivery group facility group facility facility p- p- p- p- b se value b se value b se value b se value Post indicator -0.036* 0.020 0.067 0.064 0.040 0.111 -0.060** 0.030 0.046 -0.001 0.017 0.973 PBF/Post interact 0.018 0.028 0.532 0.038 0.050 0.454 0.042 0.033 0.214 -0.038 0.025 0.127 Control 1/Post interact 0.000 0.030 0.989 0.003 0.050 0.954 0.072** 0.034 0.033 -0.012 0.023 0.602 Control 2/Post interact 0.016 0.028 0.571 0.004 0.048 0.930 0.039 0.037 0.282 -0.018 0.022 0.415 N 5419 5419 5419 5419 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from multinomial logistic difference-in-differences regression models examining the effect of PBF on facility bypassing for reproductive health care. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work), household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation) and facility-level control variables at baseline (availability of electricity, piped water, and a latrine respectively at the health facility, facility open 24-hour a day, type of health facility, urban/rural and number of health workers employed at the health facility. Standard errors were clustered at the health facility level. 31 Table 13: Health care shopping for postnatal care† Postnatal care in Postnatal care in assigned treatment unassigned treatment Postnatal care in non- No postnatal care group facility group facility randomized facility p- p- p- p- b se value b se value b se value b se value Post indicator -0.059** 0.025 0.020 0.047* 0.027 0.086 -0.031** 0.016 0.046 0.005 0.010 0.582 PBF/Post interact -0.030 0.032 0.356 0.056* 0.033 0.091 0.030* 0.018 0.098 -0.018 0.014 0.193 Control 1/Post interact 0.013 0.03 0.676 -0.015 0.031 0.623 0.041** 0.019 0.027 -0.006 0.012 0.617 Control 2/Post interact 0.014 0.031 0.643 0.010 0.032 0.746 0.014 0.019 0.464 -0.006 0.013 0.617 N 5634 5634 5634 5634 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from multinomial logistic difference-in-differences regression models examining the effect of PBF on facility bypassing for reproductive health care. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work), household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation) and facility-level control variables at baseline (availability of electricity, piped water, and a latrine respectively at the health facility, facility open 24-hour a day, type of health facility, urban/rural and number of health workers employed at the health facility. Standard errors were clustered at the health facility level. Household survey utilization results Table 13 below displays the difference-in-differences regression results for the study outcomes related to health care received during pregnancy as assessed in the household survey interviewing women with recent pregnancies or birth experiences. In the tables below, the post indicator can be interpreted as the change in the outcome over the study period in the control group. Each of the interaction terms can be interpreted as the difference between the change observed in each treatment group respectively compared to the change in the control group. Table 14 below shows that overall few treatment effects were observed for study outcomes related to care during pregnancy in the household survey data. Among women who had been pregnant in the 24 months before the survey, there was an overall increase of approximately five percentage points in delivery with a skilled birth attendant. There was no difference between the control group and the PBF group, or the group that received additional financing in the change in skilled delivery over the study period. However, skilled delivery declined (-0.050*, p-value = 0.087) in the group receiving only improved supervision relative to the control group. The change between 2012 and 2015 in the percentage of women who received at least two antenatal care visits was not statistically significant in the control group. Similar to delivery care, there was no difference between the change in the control group and the PBF and additional financing groups but the change among women in the improved supervision group was less than the change observed in the full control group (-0.044**, p-value = 0.022). There was very little change over the study period in receipt of tetanus vaccine during pregnancy, and there was no difference in the change in vaccine receipt between the three treatment groups, and the control group. Postnatal care receipt increased by over ten percentage points over the study period in control group (0.105, p-value = 0.001). Compared to the control group, there was a smaller increase over time in postnatal care in the three treatment groups; however this difference was not statistically significant in the PBF group and the additional financing group. Postnatal care receipt increased by seven percentage points less in the improved supervision group (C2) than in the control group, and this difference was statistically significant (-0.070, p- value = 0.075). Results from testing the equality of coefficients show that for skilled delivery 32 the additional financing intervention outperformed the PBF group. For antenatal care, facilities with the PBF intervention performed better than those facilities with only additional supervision. To further investigate the influence if healthcare bypassing behavior, we conducted additional analyses for key indicators dividing the sample into high and low bypass strata. We specifically tested whether the effect of the treatment groups differed in catchment areas where are larger proportion of women sought care outside of their treatment group health facility at baseline. To do this, we generated a binary variable = 1 if the women went to her assigned health facility at baseline and = 0 if she did not, and collapsed the data at the health facility level taking the mean of the bypass indicator variable. We then merged onto the original file so that for each woman in the dataset we know the proportion of women in her catchment area who went to their assigned facility at baseline. Then we divided the sample into high bypassing (above the median) and low bypassing (below the median), conducted the analysis in the separate groups, and compared the coefficients on the three interaction terms between groups. We included antenatal care, skilled delivery and postnatal care in this analysis. The hypothesis behind these additional analyses is that if there are spillovers, we should see larger treatment effects in areas with low bypass rates. The only difference in the results between high and low bypassing areas was effect of additional financing was negative (-0.108) in the high bypassing group and positive in the low bypassing group (0.045), and this difference was statistically significant (p=0.058) (results shown in Appendix F). Otherwise the impacts of the different interventions tested did not vary according to whether health care bypassing behavior was high or moderate at baseline. This analysis further suggests that the bypassing behavior observed in Cameroon, while substantial, did not significantly bias our impact measures. Table 14: Coverage of reproductive health services among women who were pregnant in the previous 24 months† At least two ANC Tetanus vaccine Skilled delivery visits during pregnancy Postnatal care β se β se β se β se Post indicator 0.053*** 0.019 0.022 0.014 0.001 0.019 0.105*** 0.031 PBF/Post interact -0.043 0.028 0.010 0.02 0.024 0.023 -0.029 0.041 Control 1/Post interact 0.020 0.032 -0.024 0.019 0.003 0.025 -0.019 0.041 Control 2/Post interact -0.050* 0.029 -0.044** 0.019 0.01 0.023 -0.070* 0.039 p-value PBF vs. C1 0.055 0.111 0.369 0.798 p-value PBF vs. C2 0.828 0.010 0.520 0.277 p-value PBF vs. C3 0.117 0.617 0.306 0.048 N 5858 5974 5975 5966 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on reproductive health service use among female respondents included in the household survey who had been pregnant in the previous 24 months. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Standard errors were clustered at the health facility level. Non-pregnant fertile women of reproductive age (15 – 49 years) with a current sexual partner were asked whether they were using any form on contraception in the household survey. Modern contraception included the intrauterine device, injectables, implants, oral pills, diaphragm, foam/jelly & lactational amenorrhea. The percentage of women of reproductive age who used any form of modern contraception, excluding condoms, did not increase between 2012 33 and 2015 in the control group (Table 15). The changes observed in the treatment groups did not differ statistically from the change in the control group, and there was no difference between the treatment groups. Table 15: Use of modern contraception among women of reproductive age† Modern contraception β se Post indicator 0.002 0.044 PBF/Post interact -0.037 0.054 Control 1/Post interact -0.054 0.055 Control 2/Post interact 0.000 0.053 p-value PBF vs. C1 0.731 p-value PBF vs. C2 0.429 p-value PBF vs. C3 0.486 N 4498 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on modern contraceptive use among female respondents of reproductive age (15 – 49) included in the household survey. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Standard errors were clustered at the health facility level. Mothers or primary care givers of all children under five years of age were asked about their child’s vaccination history. For all children with a vaccine card, study enumerators recorded all documented vaccinations and their respective receipt dates. Mothers/primary caretakers were also asked to report any vaccinations that were not recorded in the vaccine card. For these questions, enumerators asked a separate question for each vaccine type that referenced the vaccine name and also gave an indication of its method of administration (i.e. for polio “that is drops in the mouth�) as a guide for respondents. Only children between 12 – 23 months of age were included in these analyses. Both outcomes include the following vaccines: oral polio vaccine, yellow fever, diphtheria, tetanus, and whooping cough (DTC), measles, and Bacillus Calmette–Guérin (BCG). Table 16 shows that among children with a vaccine card, there was an almost 13 percentage point increase in full vaccination over the study period (0.127, p-value = 0.080). In the PBF group, there was a further 17 percentage point increase in full vaccination (0.170, p-value = 0.076). There was no difference between the control group, and the additional funding and additional supervision groups in full vaccination as documented in vaccination records. The second vaccination outcome included both documented vaccine receipt as well as any self-reported vaccines. This outcome displayed a similarly large increase in the control group (0.108, p-value = 0.039). Additionally, there was a further 16.4 percentage point increase in full vaccination over the study period in the PBF group (p-value = 0.019). Finally, neither the additional funding, nor the additional supervision group showed an increase in full vaccination beyond the increase in the control group. The p-values directly comparing the PBF group with the other groups (C1 and C2) further indicate that the PBF group outperformed the three other study groups. Table 16: Full vaccination coverage among children between 12 - 23 months of age† Fully vaccinated documented by Fully vaccinated by vaccine card or vaccine card self-report β se β se Post indicator 0.127* 0.072 0.108** 0.052 34 PBF/Post interact 0.170* 0.095 0.164** 0.069 Control 1/Post interact -0.054 0.092 -0.015 0.065 Control 2/Post interact 0.018 0.092 0.029 0.073 p-value PBF vs. C1 0.009 0.003 p-value PBF vs. C2 0.075 0.052 p-value PBF vs. C3 0.076 0.019 N 1569 2448 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on child vaccination among children (12 – 23 months) included in the household survey. Regression model adjusted for individual (age, father in the household, religion, ethnicity) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Standard errors were clustered at the health facility level. The next two outcomes assessed from the household survey were growth monitoring in the month before the survey, and having slept under a bednet the night before the survey. Both outcomes were assessed among children under five years of age; however, children who were less than 12 months old were not included in the growth monitoring analysis. There was no change in growth monitoring during the study period in the control group. Similarly growth monitoring did not increase in the PBF and the additional supervision groups. However, growth monitoring increased by approximately 3 percentage points more in the group that received additional funding (0.031*, p-value = 0.071) than the control group. The proportion of children who slept under a bednet the night before the survey declined by almost 19 percentage points during the study period (-0.186, p-value = 0.000). A similar decline was observed in all the treatment groups as shown by the small and non-statistically significant coefficients on the interaction terms. Results from testing the equality of coefficients showed that the additional financing group performed better than the PBF group in growth monitoring. It should be noted that neither growth monitoring nor bednet distribution were included in the package of services incentivized in the PBF program. Table 17: Growth monitoring in the last month among children ages 12 - 59, and having slept under a bednet during the previous night among all children under 5 years of age† Growth monitoring in the last Slept under a bednet month β se β se Post indicator -0.014 0.013 -0.186*** 0.026 PBF/Post interact -0.002 0.017 0.005 0.044 Control 1/Post interact 0.031* 0.017 0.008 0.039 Control 2/Post interact 0.022 0.019 0.013 0.036 p-value PBF vs. C1 0.047 0.941 p-value PBF vs. C2 0.215 0.848 p-value PBF vs. C3 0.930 0.913 N 7055 9995 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on growth monitoring in the last month among children (12 – 59 months), and on having slept under a bednet the night before the survey among children under 5 years of age included in the household survey. Regression model adjusted for individual (age, father in the household, religion, ethnicity) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Standard errors were clustered at the health facility level. 35 Anthropometrics The height and weight of all children under 5 years of age was recorded during both baseline and endline surveys. Though nutrition indicators were not incentivized in the Cameroon PBF program until 2015, we used the height and weight data collected in the household survey to assess the effect of PBF on child nutrition outcomes. Using the WHO child growth standard’s Stata package, we calculated a z-score for child height-for-age, weight-for-age, and weight-for- height. These z-scores represent the location where each child falls according to international standards defining healthy child development. Using these scores we calculated the prevalence of stunting (height-for-age of less than -2 standard deviations from the mean), underweight (weight-for-age of less than -2 standard deviations from the mean) and wasting (weight-for-age of less than -3 standard deviations from the mean). As shown in table 18, below, there was no difference between the changes in prevalence of stunting, underweight, and wasting in the treatment groups and the control groups. While there was no change over time in stunting and underweight, the prevalence of wasting increased at endline compared to baseline. Table 18: Anthropometric results† Stunting Underweight Wasted b se b se b se Post indicator -0.013 0.026 -0.007 0.022 0.052** 0.021 PBF/Post interact 0.021 0.033 0.045 0.029 -0.006 0.028 Control 1/Post interact 0.014 0.038 0.038 0.032 -0.033 0.03 Control 2/Post interact 0.05 0.035 0.017 0.029 -0.03 0.027 p-value PBF vs. C1 0.838 0.812 0.338 p-value PBF vs. C2 0.352 0.274 0.322 p-value PBF vs. C3 0.526 0.114 0.836 N 8327 8565 8177 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on child anthropometric outcomes (stunting, underweight and wasting). Regression model adjusted for individual (age, father in the household, religion, ethnicity) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Extreme z-scores either below -6 standard deviations from the mean, or above 6 standard deviations from the mean were removed before analysis. Standard errors were clustered at the health facility level. Facility survey utilization results This section describes the results of PBF on health services provision as recorded in facilities registers. To assess the reliability of these data, we examined the health service counter- verification data that was collected routinely as part of the PBF program design through community client satisfaction surveys. Health service verification took place in all PBF health facilities, as well as in health facilities in control groups C1 (additional financing) and C2 (additional supervision). Thirty-five patients were sampled for 7 health service categories each quarter. The figure below shows the percentage of patients who were reported by health facilities to the PBF verification terms, who were later confirmed to have received health services at the health facility. During most quarters of the three year study period in all three study regions, over 80% of reported patients were confirmed. The trend in confirmed patients increased slightly over time in North-West and East, with confirmation rates above 85% in all three regions during the final year of the study. Though we find reassuringly high quality data among the treatment group health facilities, it is possible that facilities in the full control group had less incentive to keep records of all services provided. Given that we did not verify the 36 health service data collected from the full control group, we cannot investigate this possibility. Therefore, the possibility of incomplete reporting in the full control remains a limitation of this analysis. Figure 8: Percent of reported patients confirmed during verification 100 98 94 89 83 86 80 81 Percent (%) 60 40 Sept. Dec. Feb. May. Sept. Dec. Mar. Jun. Sept. Dec. Mar. Jun. 2012 2012 2013 2013 2013 2013 2014 2014 2014 2014 2015 2015 North-West Est South-West The tables below display health service utilization results as assessed in the facility survey. Facility level provision of health services in the six months before the survey took place was assessed using patient registers from study health facilities. All data was collected at the monthly level; therefore, the interaction term coefficients represent differences between groups in the change in monthly services provided. Table 19 below displays results from health facility provision of pregnancy services. Provision of skilled delivery did not increase in the control group during the study period. Relative to the control there were no statistical differences in mean monthly provision of skilled delivery in the full PBF and the additional supervision group; however, there was a relative increase of approximately 2 monthly deliveries in the additional supervision group (additional financing 1.855, p-value = 0.071). The overall change between 2012 and 2015 in antenatal care provision in study health facilities was positive, but was not statistically significant. Relative to the control group there was an increase in antenatal care in all three treatment groups; however, none of these differences were statistically significant. Comparing the six-months before the baseline, and the six months before the endline survey, provision of tetanus toxoid vaccine declined by a monthly average of almost 17 vaccinations in control facilities each month. Compared to the change observed in the control group, there was a positive and statistically significant difference in the PBF and additional financing groups (PBF 21.521, p-value = 0.001; additional financing 15.989, p-value = 0.014). Compared to control facilities, facilities with additional supervision provided on average approximately nine more tetanus vaccines monthly to pregnant women, but this difference was not statistically significant. Like tetanus vaccine, there was a statistically significant decline in postnatal care provision in the control group over the study period. There were on average approximately four fewer monthly postnatal care visits provided in control health facilities at endline compared to baseline. Though the interaction term was positive, there was no statistical difference between the additional supervision group and the control group. However, the change in monthly provision of postnatal care in facilities in the PBF group and the additional financing group was greater than the change in the control group (PBF 4.309, p-value = 0.059, additional financing 5.513, p-value = 0.016). 37 Table 19: Provision of reproductive health services† Tetanus vaccine Skilled delivery ANC during pregnancy Postnatal care β se β se β se β se Post indicator 0.514 0.798 3.213 3.932 -16.881*** 5.563 -3.802* 2.175 PBF/Post interact 1.374 1.011 3.007 7.641 21.521*** 6.145 4.309* 2.269 Control 1/Post interact 1.855* 1.021 1.545 5.407 15.989** 6.45 5.513** 2.262 Control 2/Post interact 0.047 1.358 3.993 5.441 8.707 7.692 3.515 2.499 p-value PBF vs. C1 0.581 0.841 0.183 0.19 p-value PBF vs. C2 0.289 0.894 0.031 0.57 p-value PBF vs. C3 0.694 0.651 0.001 0.059 Baseline mean in C3 7.76 20.57 32.84 10.22 N 2182 2220 2220 2220 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on reproductive health service provision reported in facility registers. Regression models adjusted for facility controls (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 20 below presents the results of the effect of PBF on modern contraception. Modern contraception included women provided contraceptive implants, injectables, oral contraceptive pills, and the intrauterine device. There was little change in the control facilities in monthly modern contraceptive provision. In the PBF group, there was an increase of just over nine women each month on top of the small change in the control group who were provided modern family planning (9.240, p-value < 0.001). The change in modern contraceptive delivery in the additional financing group is greater than the small change in the control group with approximately six more women per month provided family planning (5.794, p-value = 0.001) in each health facility. The change is the additional supervision group was not statistically different from the change in the control group. This strong effect of PBF and additional financing on family planning in the facility register data differs from the null finding we found in the household data. The difference in the findings between data sources could be explained by several factors. One possibility is that the household survey oversampled recently pregnant women (because having a least one recently pregnant women in the household was an inclusion criteria). It is possible that the need for modern family planning is less important in this population of women among whom many have recently shown a willingness to have children. In addition, there is some evidence that women might have been uncomfortable disclosing their use of family planning in the household survey. For example, we find that 47% of women report that their husbands are against use of FP to avoid pregnancy, and these women may not have been comfortable discussing family planning in their household. Table 20: Provision of modern contraception† 38 Modern contraception β se Post indicator 0.679 1.002 PBF/Post interact 9.240*** 2.529 Control 1/Post interact 5.794*** 1.746 Control 2/Post interact 3.321 2.061 p-value PBF vs. C1 0.205 p-value PBF vs. C2 0.046 p-value PBF vs. C3 <0.001 Baseline mean in C3 3.02 N 2220 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on modern contraception provision reported in facility registers. Regression models adjusted for facility controls (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 21 displays the results of the impact of PBF on provision of childhood vaccines. In general vaccine provision in the six months before the baseline declined as compared to provision in the six months before the endline, as demonstrated by the negative and statistically significant coefficients on the post indicators for all three vaccine outcomes in the table below. This decline has been explained by Ministry of Health officials as resulting from country level stock-outs during the months before the endline. Despite these large declines, the facility results for vaccine provision may not be inconsistent with the household vaccine coverage results, which showed large increases over time, and a particularly large change in the PBF group. According to the recommended vaccine schedule, most childhood vaccines should be received between birth and 12 months of age. To avoid including infants who, due to their age, should not yet have finished their vaccine schedule, household level coverage only included children between 12 – 23 months old. Consequently, the children included in the household data would have received vaccinations prior to the months covered in the facility level data at endline (i.e. before the vaccine stock-out). Both sets of results show that the PBF group, and to a lesser degree, the additional financing group, performed much better than the control group. Therefore, the findings from the household and facility data showing large and statistically significant differences between groups are consistent, and we believe that the level changes can be explained by a country level shock affecting vaccine availability that affected the facility, but not the household data. Facility-level provision of the third dose of polio vaccine decreased by approximately five vaccines per month in the control group. There was an increase of 4.583 final polio vaccinations relative to the control group in the PBF group, and this difference was statistically significant (p=0.035). The coefficient on the interaction term was also positive in the additional financing group but the difference was not statistically significant, and there was no difference between the control and the additional supervision groups. There was a large and statistically significant decline in meningitis vaccination provision over the study period (-45.970, p-value < 0.001). Of the three treatment groups, only the change in the additional financing group was statistically different from the control group (21.931, p-value 0.050). Finally, in the control group measles vaccine provision declined by an average of approximately four children per month over the study period. There was no difference between the change observed in the control group, and the changes in the treatment groups. Table 21: Provision of childhood vaccines† 39 Third dose of polio Meningitis vaccine Measles vaccine vaccine β se β se β se Post indicator -5.280*** 1.983 -45.970*** 9.769 -3.736* 2.249 PBF/Post interact 4.583** 2.162 19.041 13.471 3.758 2.552 Control 1/Post interact 2.765 2.389 21.931* 11.131 1.892 2.700 Control 2/Post interact 1.081 3.953 8.47 13.547 -0.740 3.546 p-value PBF vs. C1 0.252 0.753 0.337 p-value PBF vs. C2 0.322 0.387 0.135 p-value PBF vs. C3 0.035 0.159 0.143 Baseline mean in C3 23.90 46.65 20.90 N 2220 2220 2220 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on provision of vaccines to children reported in facility registers. Regression models adjusted for facility controls (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 22: Provision of HIV services† HIV testing PMTCT ART β se β se β se Post indicator 4.239 3.031 -3.552 3.323 1.021* 0.609 PBF/Post interact 61.115*** 17.817 2.084 4.011 -1.455 0.888 Control 1/Post interact 51.466*** 13.668 2.372 3.189 -0.671 0.573 Control 2/Post interact 6.596 5.757 1.648 3.156 -0.681 0.595 p-value PBF vs. C1 0.656 0.905 0.235 p-value PBF vs. C2 0.003 0.851 0.252 p-value PBF vs. C3 0.001 0.604 0.103 Baseline mean in C3 9.98 9.86 0.010 N 2220 2220 2220 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on provision of vaccines to children reported in facility registers. Regression models adjusted for facility controls (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Facility register data also contained data documenting facility provision of HIV-related services. We found a large and statistically significant effect of both PBF and additional financing on HIV testing. An average of 61 more patients were tested for HIV in PBF facilities than control facilities, and 51 more patients were tested monthly in the additional financing arm compared to the control. There was very little change in HIV testing in the additional supervision group, and the effect of PBF was greater than the effect of additional supervision. Though there was a small increase in PMTCT in all of the treatment groups relative to the full control, none of these differences were statistically significant, and there was no different between the effect of PBF and the other treatment groups. Similarly, there was no difference in the change in ART provision between the treatment groups and the full control. 40 Health care spending All household members were asked if they had been sick in the four weeks before the survey. Mothers or caregivers responded on behalf of household children. All respondents who had been sick were asked if they had gone to any health facility, health personnel or traditional healer to seek care for this illness. Respondents were then asked how much the household spent out-of- pocket for the treatment of this recent illness. Table 23 below presents the results for several different types of reported spending. Neither the change in the control group on spending for official provider fees, nor the differences between the treatment groups and the control group were statistically significant. There was an increase of approximately 2052 CFA ($3.31) in the amount of unofficial provider fees paid in the control group between 2012 and 2015. The difference between the PBF group and the control group in the change over the study period was -2254 CFA ($3.64), and this difference was statistically significant. Relative to the control group, unofficial payments also declined in the additional financing group and the additional supervision group; however, these differences were not statistically significant. There was a non-significant increase of approximately 1048.64 CFA ($1.69) in laboratory and x-ray fees over time in the control group. Compared to the control group, laboratory and x-rays fees declined by 1473.44 CFA ($2.38), and this difference was statistically significant (p-value = 0.060). Finally, transportation fees did not change between 2012 and 2015 in the control group. There was a statistically significant difference between the control group, and the additional financing group with a decrease of 495.14 or $0.80. Comparing the differences between PBF and the other treatment groups, we find that the decline in lab and x-ray fees in the PBF group was greater than the decline observed in C1. Table 23: Health care spending as reported in household data† Official provider Unofficial provider fee fee Lab and x-ray fees Transportation fees β se β se β se β se Post indicator 1811.58 1475.25 2052.12* 1057.18 1048.64 711.54 123.03 201.09 PBF/Post interact -1495.83 1538.26 -2254.12* 1305.64 -1473.44* 779.6 -455.41 288.36 Control 1/Post interact -334.73 1506.73 -2736.04 1778.02 -521.02 868.01 -495.14** 241.38 Control 2/Post interact -1378.05 3969.75 -1422.67 1244.33 -639.27 885 -368.79 236.41 p-value PBF vs. C1 0.191 0.750 0.051 0.880 p-value PBF vs. C2 0.974 0.392 0.128 0.732 p-value PBF vs. C3 0.332 0.086 0.060 0.116 N 2374 2261 2292 2365 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on health care spending in the last 4 weeks among respondents in the household survey. Regression model adjusted for individual (age, sex) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation). Standard errors were clustered at the health facility level. Health spending was also assessed in the exit interview surveys conducted among women who had received prenatal care, and the caregivers of children under five years of age who visited the health facility for a child health consultation. Tables 24, and 25 below display these results, even though reports by patients about health spending might be more reliable when given in the privacy of the household (household survey) than during exit interviews which take place in or close to the health facility compound, especially for reports about unofficial payments to providers. There were no changes over time, and no statistically significant differences between 41 groups in unofficial provider fees, medicine fees, and total fees for antenatal care. The change in official provider fees for antenatal care was -1025.34 CFA ($1.68) lower in the PBF group compared to the control group, and this difference was statistically significant (p-value = 0.083). Relative to the control group, the change in the additional financing group in official provider fees was 1824.81 CFA ($2.98) higher (p-value = 0.038). Spending on official provider fees was significantly lower in the PBF groups than in the three other groups. Additionally, there was a statistically significant increase in the additional financing group and the additional supervision group in total fees for antenatal care. There were no changes over time, or differences between groups in any of the spending outcomes (official provider fee, unofficial provider fee, medicine fees, total fees) for child health consultations. Table 24: Health spending for ANC reported from ANC exit interviews† Unofficial Official provider fee provider fee Medicines fees Total fees β se β se β se β se Post indicator 472.17 438.97 -217.86 199.56 -695.08 647.31 319.34 1608.44 PBF/Post interact -1025.34* 585.71 136.57 231.57 701.7 708.71 2501.29 2637.04 Control 1/Post interact 1824.81** 867.46 312.76 278.58 1260.72 825.5 4445.44* 2460.9 Control 2/Post interact -67.28 483.7 203.69 191.43 2374.42 1813.13 5178.78** 2560.66 p-value PBF vs. C1 0.00 0.37 0.39 0.49 p-value PBF vs. C2 0.02 0.55 0.34 0.35 p-value PBF vs. C3 0.08 0.56 0.32 0.34 Baseline mean in C3 604.91 232.79 1881.96 5239.51 N 725 730 652 724 † Results from difference-in-differences regression models examining the effect of PBF on spending for antenatal care among respondents to antenatal care exit interviews. Regression model adjusted for individual (age, literacy, education level, marital status) and facility variables (type of health facility, urban/rural). Standard errors were clustered at the health facility level. Standard errors were clustered at the health facility level. Table 25: Health spending for child health consultations reported from consultations of children under 5 years of age† Official provider Unofficial provider fee fee Medicines fees Total fees β se β se β se β se Post indicator -14.37 57.05 -34.34 20.87 227.67 559.02 403.6 887.2 PBF/Post interact 79.17 133.41 36.22 26.31 679.38 729.58 1545.01 1282.14 Control 1/Post interact 43.76 113.03 12.05 32.24 442.13 819.08 636 1283.09 Control 2/Post interact 53.25 89.29 -22.76 60.83 14.41 786.8 731.23 1105.73 p-value PBF vs. C1 0.81 0.4 0.76 0.48 p-value PBF vs. C2 0.86 0.31 0.41 0.50 p-value PBF vs. C3 0.55 0.17 0.35 0.23 Baseline mean in C3 286.79 85.57 2105.00 2921.51 N 613 612 556 609 † Results from difference-in-differences regression models examining the effect of PBF on spending for child health consultations among respondents to child health care exit interviews. Regression model adjusted for individual (age, literacy, education level, marital status) and facility variables (type of health facility, urban/rural). Standard errors were clustered at the health facility level. Standard errors were clustered at the health facility level. 42 Patient satisfaction Satisfaction with antenatal care Table 26, below, provides an overview of participants in the ANC exit interviews at baseline. The average age of respondents was just over 25 years, 75 percent of respondents were married at the time of the interview, and 72 percent were literate. As shown in Table 26 below, the intervention arms were well balanced across all of these dimensions. There is, however, evidence of differences in education. In particular, women attending facilities in the additional financing arm were significantly more likely to have attended higher education than were women attending other facilities (p<0.05). Table 26: Sample characteristics of women included in the antenatal care exit interviews as baseline Mean Mean Mean Mean Mean p-value p-value p-value p-value T1 C1 C2 C3 total T1/C3 C1/C3 C2/C3 F-test N Age 25.82 24.95 25.26 24.93 25.25 0.36 0.99 0.77 0.80 258 Currently married 0.74 0.81 0.83 0.79 0.79 0.56 0.79 0.57 0.67 258 Literate 0.82 0.72 0.70 0.65 0.72 0.03 0.40 0.58 0.19 256 No education 0.08 0.13 0.10 0.10 0.10 0.65 0.59 0.95 0.80 258 Primary education 0.55 0.39 0.51 0.43 0.47 0.18 0.66 0.36 0.26 258 Secondary education 0.33 0.27 0.33 0.38 0.33 0.61 0.22 0.60 0.69 258 Secondary education level 2 0.05 0.06 0.04 0.08 0.06 0.40 0.71 0.36 0.77 258 Higher education 0.00 0.15 0.01 0.02 0.04 0.30 0.01 0.93 0.00 258 Women were asked a series of twelve questions related to their satisfaction with individual elements of their visits including, for example, their satisfaction with costs, wait times, and health worker communication. For each, a statement was read, and women were asked if they agreed, were neutral, or they disagreed. Binary variables were created by coding responses as “1� if a woman agreed, and “0� otherwise. Overall satisfaction scores were calculated by averaging over these twelve components. An overall score of “1� indicates that a woman agreed with all twelve questions, while a score of “0� indicates that she either disagreed or was neutral on all twelve questions. The impact of the interventions on overall satisfaction are shown in Table 27 below. There is no indication that satisfaction changed over time in the control group (β = 0.006, p = 0.847). Relative to the pure control, the PBF group was associated with an 8.6 percentage point increase in satisfaction (p = 0.077). The results suggest a stronger effect in the full PBF than in the additional supervision group (10.5-percentage point increase). 43 Table 27: Overall satisfaction score for antenatal care reported during facility exit interviews† Overall ANC satisfaction score β se p-value Post indicator 0.006 0.034 0.847 PBF/Post interact 0.086* 0.048 0.077 Control 1/Post interact 0.051 0.044 0.246 Control 2/Post interact -0.019 0.049 0.698 p-value PBF vs. C1 0.419 p-value PBF vs. C2 0.036 p-value PBF vs. C3 0.077 Baseline mean in C3 0.853 N 730 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression model examining the effect of PBF on the overall satisfaction score for antenatal care reported by patients during facility exit interviews. Regression model adjusted for individual (age, literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Tables 28 – 30 show the breakdown of the twelve individual components of satisfaction. Regression results indicate the percentage point increase or decrease in probability that a respondent in a given intervention group will agree with a statement, relative to the pure control, after adjusting for individual-level indicators (age, literacy, marital status, and education level) and facility-level indicators (type of health facility public/private/religious, urban/rural). Table 28 focuses on the costs associated with care. For the first three dependent variables reporting whether women found the fees reasonable, while few of the results meet the 10 percent cutoff for statistical significance, women in the PBF group are consistently more likely to say that fees are reasonable than either the full control or partial treatment groups. The only statistically significant difference among these fee-related variables was that women in the PBF group were more likely to agree that medicine fees were reasonable than women in the additional financing and improved supervision groups. The final column of Table 28 relates to informal payments. There were no differences between groups in the likelihood that women agreed with the statement that health workers did not ask for additional presents or payments. Table 28: Satisfaction with payments for antenatal care reported during facility exit interviews† Reasonable Reasonable lab Reasonable No additional registration fees fees medicine fees payment β se β se β se β se Post indicator 0.055 0.097 0.074 0.075 0.088 0.09 0.01 0.06 PBF/Post interact 0.037 0.128 0.154 0.113 0.190 0.134 -0.043 0.078 Control 1/Post -0.051 0.142 -0.051 0.129 -0.069 0.121 0.067 0.092 interact Control 2/Post -0.085 0.127 -0.027 0.127 -0.127 0.118 -0.02 0.087 interact p-value PBF vs. C1 0.523 0.129 0.044 0.230 p-value PBF vs. C2 0.309 0.185 0.015 0.776 p-value PBF vs. C3 0.774 0.176 0.158 0.586 Baseline mean in C3 0.804 0.782 0.754 0.885 N 669 665 689 723 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with payment for antenatal care reported by patients during facility exit interviews. Regression model adjusted for 44 individual (age, literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 29 focuses on women’s satisfaction with health facility characteristics. PBF is associated with a large and statistically significant 24.1-percentage point difference from the control group (p<0.05) in women’s satisfaction with the health facility cleanliness and facility hours. Focusing on reported facility cleanliness, women in the PBF and the additional financing group (β=0.228, p<0.05) both reported significantly higher levels of agreement than in the pure control group, although these scores were not significantly different from one another. Although not meeting the 10 percent cut-off, this pattern was also seen in response to wait times, where the PBF group was associated with 16.1-percentage point increase over the pure control, compared to a decrease of 5.5-percentage points in the supervision only group. The pattern is slightly different in response to the question on privacy – while the additional supervision group continued to underperform relative to the pure control, the point estimate of the difference between the PBF and pure control is almost 0, while the additional financing group reported the highest rate of satisfaction. However, none of these estimates achieve statistical significance. The results for the adequacy of hours are consistent with the pattern observed earlier. Compared to the control group, PBF results in a large and statistically significant 15.4-percentage point increase satisfaction with the facility’s hours, while the additional supervision is associated with a non- significant reduction in satisfaction relative to the pure control. As indicated by the p-value on PBF versus the improved supervision, PBF performed consistently and significantly better than the improved supervision group on facility cleanliness, and adequacy of hours. Table 29: Satisfaction with health facility characteristics during antenatal care reported during facility exit interview† Clean health Reasonable wait Enough privacy Adequate hours facility time during visit β se β se β se β se Post indicator -0.045 0.075 -0.009 0.079 -0.025 0.061 -0.026 0.041 PBF/Post interact 0.241** 0.111 0.161 0.115 0.042 0.086 0.154** 0.071 Control 1/Post 0.228** 0.106 0.014 0.129 0.149 0.093 0.032 0.055 interact Control 2/Post 0.002 0.111 -0.055 0.134 -0.010 0.098 -0.079 0.062 interact p-value PBF vs. C1 0.903 0.269 0.254 0.073 p-value PBF vs. C2 0.040 0.127 0.605 0.002 p-value PBF vs. C3 0.032 0.163 0.629 0.033 Baseline mean in C3 0 .787 0 .738 0 .902 0.900 N 730 727 728 724 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with health facility characteristics reported by patients during facility exit interviews. Regression model adjusted for individual (age, literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 30 provides an overview of women’s reported satisfaction with health worker characteristics. Women attending facilities receiving the full PBF intervention reported significantly higher levels of satisfaction with health worker communication than did women attending control clinics (β=0.106, p<0.05), but there was no evidence of an impact of PBF on the courteousness of health staff, time with health workers, or the ease of getting prescribed medicines. Women attending facilities receiving additional financing reported significantly higher levels of satisfaction with the amount of time they spent with health workers than women 45 in control clinics (β=0.139, p<0.1). This increase in the additional financing group was greater than the change in the PBF group (which was negative relative to the control group). Table 30: Satisfaction with health worker characteristics during antenatal care reported during facility exit interview† Courteous Good health Sufficient visit Easy to get health staff worker time with health prescribed communication worker medicines β se β se β se β se Post indicator -0.041 0.036 -0.031 0.024 0.055 0.052 0.018 0.068 PBF/Post interact 0.037 0.063 0.106** 0.050 -0.045 0.074 0.000 0.079 Control 1/Post interact 0.07 0.053 0.039 0.052 0.139* 0.080 0.041 0.085 Control 2/Post interact 0.019 0.051 0.101 0.077 0.030 0.101 -0.040 0.075 p-value PBF vs. C1 0.599 0.308 0.026 0.488 p-value PBF vs. C2 0.783 0.961 0.443 0.402 p-value PBF vs. C3 0.556 0.038 0.544 0.997 Baseline mean in C3 0 .967 0 .951 0 .852 0 .883 N 730 725 728 716 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with health worker characteristics during antenatal care reported by patients during facility exit interviews. Regression model adjusted for individual (age, literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Satisfaction with child health consultations (< 5 years old) Table 31, below, provides an overview of participants in the child health consultation exit interviews at baseline. The mean age of children attending facilities in the PBF arm was 23.7 months, which is older, on average, than those attending facilities in other arms (p<0.05). The youngest group was those attending facilities in the additional financing arm (mean age: 14.2 months). The intervention arms are well balanced on gender, with females accounting for approximately 51 percent of children attending the facilities. Looking at the caretaker characteristics, those in the control group were more likely to be single (16 percent, compared to a sample mean of 20 percent), while those in the PBF group were least likely to be married (60 percent, compared to a sample mean of 73 percent). Few (3 percent, overall) were divorced or widowed, and the treatment arms were well balanced on literacy and education. Approximately 75 percent of the caretakers were literate (low: 69 percent in the improved supervision arm; high: 77 percent in the additional financing and pure control arms). The majority of women had some education, with most stopping during primary school (39 percent) or secondary level 1 (30 percent). 46 Table 31: Sample characteristics of children and mothers included in child health consultation exit interviews at baseline Mean Mean Mean Mean Mean p-value p-value p-value p-value T1 C1 C2 C3 total T1/C3 C1/C3 C2/C3 F-test N Child characteristics Age in months 23.66 14.15 22.45 18.00 20.01 0.09 0.24 0.16 0.03 185 Female 0.58 0.40 0.54 0.49 0.51 0.35 0.41 0.63 0.40 188 Caretaker characteristics Single 0.35 0.20 0.26 0.16 0.25 0.03 0.63 0.22 0.14 187 Currently married 0.60 0.77 0.74 0.80 0.73 0.03 0.75 0.47 0.15 187 Divorced or widowed 0.04 0.03 0.00 0.04 0.03 0.97 0.78 0.14 0.53 187 Literate 0.75 0.77 0.69 0.77 0.74 0.78 0.98 0.30 0.72 190 No education 0.06 0.11 0.17 0.08 0.11 0.74 0.59 0.18 0.34 187 Primary education 0.38 0.37 0.43 0.38 0.39 0.96 0.94 0.63 0.94 187 Secondary education level 1 0.27 0.26 0.30 0.38 0.30 0.25 0.24 0.37 0.58 187 Secondary education level 2 0.21 0.17 0.07 0.12 0.14 0.24 0.50 0.43 0.24 187 Higher education 0.08 0.09 0.04 0.04 0.06 0.37 0.38 0.94 0.63 187 Caretakers were asked a series of twelve questions related to their satisfaction with individual elements of visits. These questions were the same as those asked following antenatal visits and included, for example, their satisfaction with costs, wait times, and health worker communication. For each, a statement was read and women were asked if they agreed, were neutral, or they disagreed. Binary variables were created by coding responses as “1� if a woman agreed, and “0� otherwise. Overall satisfaction scores were calculated by averaging over these twelve components. An overall score of “1� indicates that a woman agreed with all twelve questions, while a score of “0� indicates that she either disagreed or was neutral on all twelve questions. Results are shown in Table x below. We find evidence that PBF had a positive impact on overall satisfaction with child health services (Table 32). Relative to the pure control, the PBF was associated with a statistically significant 9.9-percentage point increase in satisfaction (p<0.05). As was the case with the ANC exit interviews, we find a stronger effect in the full PBF than in the additional financing group (5.4 percentage point increase relative to full control) or supervision group (2.2-percentage point increase), and neither of the partial treatments achieves statistical significance at the 10 percent cutoff. While PBF and the additional financing group are not statistically significantly different from one another, the 7.7-percentage point difference in reported satisfaction between the PBF and the enhanced supervision (C2) group is statistically significant (p<0.10). 47 Table 32: Overall satisfaction score for child health consultations reported during facility exit interviews† Overall child consultation satisfaction score β se p-value Post indicator -0.036 0.026 0.123 PBF/Post interact 0.099*** 0.037 0.013 Control 1/Post interact 0.054 0.04 0.361 Control 2/Post interact 0.022 0.045 0.644 p-value PBF vs. C1 0.280 p-value PBF vs. C2 0.092 p-value PBF vs. C3 0.009 Baseline mean in C3 0 .881 N 614 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression model examining the effect of PBF on the overall satisfaction score for child health consultations reported by mothers during facility exit interviews. Regression model adjusted for individual (child age, child sex, maternal literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Tables 33 – 35 show the breakdown of the twelve individual components of satisfaction. Point estimates indicate the percentage point increase or decrease in probability that a respondent in a given intervention group will agree with a given statement, relative to the pure control, after adjusting for individual-level indicators (age, literacy, marital status, and education level) and facility-level indicators (availability of electricity, availability of piped water, availability of latrine, facility open 24 hours, type of health facility, urban/rural status, and number of health workers employed at the facility). Table 35 focuses on the costs associated with care. All three groups are associated with statistically significantly higher satisfaction with laboratory fees relative to the pure control. However, they are not significantly different from one another. The additional financing control had the highest (non-significant 11.1-percentage point) estimated impact on satisfaction with medicine costs; however, none of the differences between the control and treatment groups were statistically significant for this outcome. There was no difference between any of the intervention arms, and the control group in satisfaction with registration fees, and informal payments. Table 33: Satisfaction with payments for child health consultations reported during facility exit interviews† Reasonable Reasonable lab fees Reasonable No additional registration fees medicine fees payment β se β se β se β se Post indicator -0.054 0.061 -0.143 0.118 -0.045 0.067 -0.004 0.055 PBF/Post interact 0.112 0.101 0.347** 0.175 0.043 0.124 -0.007 0.117 Control 1/Post interact 0.019 0.076 0.331* 0.167 0.111 0.103 -0.02 0.077 Control 2/Post interact 0.074 0.08 0.420** 0.166 0.033 0.12 -0.055 0.09 p-value PBF vs. C1 0.325 0.925 0.595 0.908 p-value PBF vs. C2 0.709 0.685 0.945 0.696 p-value PBF vs. C3 0.268 0.050 0.731 0.953 Baseline mean in C3 0.957 0 .846 0 .854 0.904 N 488 369 544 605 * = p < 0.10, ** p < 0.05, *** p< 0.01 48 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with payment for child health consultations reported by mothers during facility exit interviews. Regression model adjusted for individual (child age, child sex, maternal literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). As shown in Table 34, PBF is associated with a large and statistically significant impact on satisfaction with the health facility cleanliness (β=0.227, p=0.090). Neither the additional financing nor the additional supervision intervention groups (C1 and C2) performed better on cleanliness at endline than they did at baseline. Focusing on waiting times, all of the intervention arms appear to result in improvements over the control group, although none of the differences were statistically significant. All of the arms also resulted in increased satisfaction with the privacy at health facilities, and the very large point estimate on the PBF (33.6 percentage points) is significant at p<0.01. Satisfaction with the opening hours did not change over time in any of the treatment groups, and there were no differences between groups for this outcome. Table 34: Satisfaction with selected health facility characteristics during child health consultations reported during facility exit interviews† Clean health Reasonable wait Enough privacy Adequate hours facility time during visit β se β se β se β se Post indicator -0.141 0.099 -0.111** 0.055 -0.098 0.098 -0.046 0.052 PBF/Post interact 0.227* 0.133 0.110 0.097 0.336*** 0.124 0.085 0.068 Control 1/Post interact 0.136 0.118 0.143 0.097 0.202 0.131 0.036 0.068 Control 2/Post interact -0.049 0.131 0.021 0.100 0.093 0.115 -0.116 0.081 p-value PBF vs. C1 0.403 0.777 0.256 0.427 p-value PBF vs. C2 0.019 0.442 0.021 0.007 p-value PBF vs. C3 0.090 0.259 0.007 0.210 Baseline mean in C3 0 .868 0 .943 0 .774 0 .942 N 612 608 612 608 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with health facility characteristics reported by mothers during facility exit interviews. Regression model adjusted for individual (child age, child sex, maternal literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 35 provides an overview of caretakers’ reported satisfaction with health worker characteristics. PBF does not appear to have had any impact on health staff courteousness (β = - 0.012), and the point estimates on the two partial treatments indicate a negative impact on health worker courteousness, relative to the pure control. All three intervention groups have negative, but not statistically significant, point estimates on satisfaction with the time spent with health workers, compared to the pure control. By contrast, all three were associated with positive but non-statistically significant impacts on the ease of getting prescribed medications. Overall, standard errors are large relative to point estimates and, across these variables, no statistically significant differences could be seen between groups, either between the pure control and the three intervention groups or between the different intervention arms themselves. 49 Table 35: Satisfaction with selected health worker characteristics during child health consultations reported during facility exit interviews† Courteous health Good health worker Sufficient visit time Easy to get staff communication with health worker prescribed medicines β se β se β se β se Post indicator 0.062 0.062 0.017 0.073 0.091 0.066 -0.002 0.059 PBF/Post interact -0.012 0.079 0.053 0.092 -0.094 0.087 0.055 0.082 Control 1/Post interact -0.080 0.077 -0.08 0.105 -0.018 0.111 0.068 0.087 Control 2/Post interact -0.101 0.082 0.055 0.103 -0.031 0.104 0.112 0.106 p-value PBF vs. C1 0.323 0.17 0.461 0.890 p-value PBF vs. C2 0.214 0.988 0.507 0.594 p-value PBF vs. C3 0.876 0.566 0.279 0.498 Baseline mean in C3 0 .887 0 .830 0 .830 0 .925 N 613 606 609 610 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on satisfaction with health worker characteristics during antenatal care reported by mothers during facility exit interviews. Regression model adjusted for individual (child age, child sex, maternal literacy, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Health worker satisfaction and motivation In all, 434 health workers were interviewed at baseline. Key characteristics are described in Table 36, below. The mean age of health workers was approximately 39 years and, on average, workers had been employed at the facility for between 4.4 (improved supervision) and 6.2 years (control; statistically different with p< 0.05). Approximately two-thirds of health workers were female, and 75 percent had received either basic or level 2 secondary education. Just under half of workers were employed by the Ministry of health, with the remaining employed by religiously affiliated health facilities (19 percent) the facility (16 percent) or other employers (20 percent). Table 36: Sample characteristics of health workers in study health facilities at baseline Mean Mean Mean Mean Mean p-value p-value p-value p-value T1 C1 C2 C3 total T1/C3 C1/C3 C2/C3 F-test N Provider age 38.72 41.31 37.18 38.67 38.96 0.98 0.13 0.30 0.11 434 Years employed at facility 5.43 4.50 4.35 6.24 5.13 0.37 0.05 0.03 0.08 428 Provider sex 0.66 0.68 0.66 0.69 0.67 0.71 0.93 0.63 0.96 434 Primary education 0.23 0.15 0.13 0.22 0.18 0.85 0.16 0.07 0.12 434 Secondary education 0.31 0.36 0.44 0.44 0.39 0.05 0.27 0.92 0.13 434 Secondary education level 2 0.40 0.41 0.34 0.30 0.36 0.13 0.10 0.54 0.30 434 Higher education 0.06 0.07 0.06 0.04 0.06 0.47 0.31 0.32 0.75 434 Employed by MOH 0.45 0.44 0.49 0.45 0.46 0.98 0.87 0.55 0.88 433 Religious employer 0.13 0.25 0.20 0.17 0.19 0.41 0.14 0.54 0.14 433 Employed by facility 0.19 0.09 0.15 0.20 0.16 0.83 0.03 0.33 0.14 433 Other employer 0.23 0.21 0.16 0.18 0.20 0.33 0.52 0.65 0.50 433 50 Health workers were read a series of statements relating to their wellbeing over the two weeks prior to the survey. These statements were taken from the World Health Organization’s (WHO) Well-being Index (Appendix G) and included, for example, “In the last two weeks, I have felt active and vigorous.� For each question, health workers were asked to indicate whether the statement described their state most of the time, more than half the time, less than half the time, only rarely, or never. For the purposes of analysis, these data were recoded into binary indicators. Responses were coded as “0� if the health worker replied half the time or less, and “1� otherwise. Thus, point estimates on the postXintervention arms indicate the percentage point increase or decrease in probability that a respondent in a given intervention group reported that a given statement was true at least half the time, relative to the pure control, after adjusting for individual-level indicators (age, sex, marital status, and education level) and facility-level indicators (availability of electricity, availability of piped water, availability of latrine, facility open 24 hours, type of health facility, urban/rural status, and number of health workers employed at the facility). Overall, the data do not provide strong evidence that PBF affected attributes included in the WHO’s wellbeing index (Table 37). Point estimates are generally small relative to standard errors, and there are no statistical differences between the pure control and the other intervention groups. However, the change in index score for three items – Active and energetic in the last 2 weeks, Refreshed and rested in the morning in the last 2 weeks, and Days filled with interesting things in the last 2 weeks was lower in the PBF group than the additional financing group. Table 37: World Health Organization well-being index† Happy and in a Calm and Active and Refreshed and Days filled good mood in relaxed in the energetic in the rested in the with interesting the last 2 last 2 weeks last 2 weeks morning in the things in the weeks last 2 weeks last 2 weeks β se β se β se β se β se Post indicator -0.025 0.057 -0.128 0.084 0.055 0.063 -0.069 0.09 0.081 0.086 PBF/Post interact 0.044 0.082 0.016 0.108 -0.117 0.074 -0.053 0.108 -0.157 0.112 Control 1/Post interact -0.009 0.087 0.094 0.114 0.022 0.079 0.134 0.113 0.062 0.116 Control 2/Post interact -0.039 0.08 0.037 0.113 -0.123 0.076 0.058 0.12 -0.096 0.106 p-value PBF vs. C1 0.569 0.451 0.025 0.037 0.047 p-value PBF vs. C2 0.332 0.835 0.906 0.254 0.528 p-value PBF vs. C3 0.592 0.885 0.117 0.624 0.163 Baseline mean in C3 0.816 0.684 0.776 0.643 0.582 N 991 991 990 991 991 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on WHO well-being index items reported by health workers. Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Health workers were also asked a series of 26 questions related to their satisfaction with working conditions. These questions covered a range of topics, including relationships with individuals within and outside of the health facility, facility infrastructure and readiness to deliver services, salary and benefits, and their overall capacity to provide high-quality health services. For each question, a score of “1� indicates satisfaction, while a “0� indicates that a health worker reported either indifference or dissatisfaction. Point estimates, thus, indicate the percentage point increase or decrease in probability that a respondent in a given intervention group is satisfied with a particular issue, relative to the pure control, after adjusting for individual-level indicators (age, sex, marital status, and education level) and facility-level 51 indicators (availability of electricity, availability of piped water, availability of latrine, facility open 24 hours, type of health facility, urban/rural status, and number of health workers employed at the facility). There is little evidence of impact on working relationships (Table 38). The additional financing arm had the largest estimated impact on the relationship between the facility and District or Ministry of Health staff, but at 12.7 percentage points, this did not meet the 10-percent cutoff for significance. The PBF point estimate for an impact on the relationship with District or Ministry of Health staff is also positive and fairly large (β=0.103), but not statistically significant. The impact on intra-facility working relationships was estimated to be negative, though not statistically significant, in all treatment groups. Satisfaction with working relationships between management and staff within the facilities declined in both the PBF and the supervision arms. Additionally, working relationships with management staff improved in C1 compared to the PBF group. There was no difference between the change in the control group and the change in the intervention groups in satisfaction with collaboration with the regional health delegation, or in the quality of the management of the health facility. Table 38: Internal and external working relationships† Working Working Working Collaboration Quality of the relationships relationships relationships with the management of with District/ with other with Regional the health Ministry of Management facility staff Health facility by the Health staff staff within the Delegation management health facility staff within the health facility β se β se β se β se β se Post indicator -0.038 0.085 0.074 0.087 0.034 0.072 0.172 0.148 0.057 0.084 PBF/Post interact 0.103 0.106 -0.029 0.104 -0.172* 0.093 -0.006 0.178 -0.089 0.123 Control 1/Post 0.127 0.109 -0.049 0.105 0.003 0.096 0.054 0.182 0.063 0.12 interact Control 2/Post -0.003 0.125 -0.129 0.106 -0.184* 0.095 -0.162 0.198 -0.057 0.129 interact p-value PBF vs. C1 0.800 0.818 0.045 0.677 0.199 p-value PBF vs. C2 0.342 0.259 0.895 0.332 0.808 p-value PBF vs. C3 0.333 0.780 0.067 0.973 0.471 Baseline mean in C3 0.793 0.763 0 .758 0.475 0.591 N 840 946 946 655 961 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on internal facility relationships, and external relationships with the District, Ministry, and Regional Health Delegation reported by health workers. Regression model adjusted for individual (age, sex, marital status, education level) and facility- level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 39, below, provides impact estimates on relationships with the communities. There is no strong evidence of an impact either on the relationships between the facility and local leaders or on health workers’ perceptions of their own status within the community. In both cases, we report a negative but not statistically significant effect of both the PBF and the improved supervision interventions. The point estimates within the financing arms are positive and, in the case of relationships with local leaders, relatively large at 9.3 percentage points, but they are not statistically significant. There was also no evidence of a difference in impact between the intervention arms. 52 Table 39: Relationships with local traditional leaders and respect in the community† The relationships between the health Your level of respect in the facility and local traditional leaders community β se β se Post indicator 0.030 0.089 0.001 0.053 PBF/Post interact -0.002 0.104 -0.034 0.065 Control 1/Post interact 0.093 0.118 0.002 0.074 Control 2/Post interact -0.074 0.112 -0.027 0.069 p-value PBF vs. C1 0.320 0.568 p-value PBF vs. C2 0.392 0.899 p-value PBF vs. C3 0.981 0.604 Baseline mean in C3 0.648 0.847 N 908 987 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on relationships with local traditional leaders and respect in the community reported by health workers. Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. By contrast, we see a large and consistent impact on health workers’ satisfaction with the quantity and quality of equipment and other supplies at health facilities, shown in Table 40, below. Both the PBF and the additional financing arms result in similarly large and highly significant improvements in these measures: an approximately 20 percentage point increase in reported satisfaction with the quantity of equipment (p<0.05), approximately 25 percentage point increase in reported satisfaction with the quality of equipment (p<0.05), and a 33 to 40 percentage point increase in satisfaction with the availability of other supplies at the health facilities within these two arms (p<0.01). By contrast, there was less evidence of an impact in the improved supervision arm. While point estimates are positive, they are not statistically significant at the 10 percent cut-off level. There is also less evidence for an effect on the quality and quantity of medicines. Although the two arms that include additional revenue – the PBF and the financing only arms – both result in positive point estimates, they do not achieve statistical significance. Table 40: Quantity and quality of health supplies, medicines, and equipment in the health facilities† Quantity of Quality of Quantity of Quality and Availability of medicine medicine equipment in physical other supplies in available in available in the health condition of the health the health the health facility equipment in the facility facility facility health facility β se β se β se β se β se Post indicator 0.092 0.081 0.070 0.067 0.032 0.060 0.022 0.071 -0.032 0.100 PBF/Post interact 0.071 0.114 0.001 0.096 0.190** 0.095 0.256** 0.109 0.404*** 0.120 Control 1/Post interact 0.176 0.111 0.050 0.096 0.210** 0.090 0.247** 0.101 0.332*** 0.121 Control 2/Post interact 0.025 0.119 -0.058 0.104 0.122 0.094 0.080 0.107 0.170 0.131 p-value PBF vs. C1 0.340 0.602 0.845 0.931 0.455 p-value PBF vs. C2 0.701 0.568 0.512 0.124 0.034 p-value PBF vs. C3 0.536 0.990 0.048 0.020 0.001 Baseline mean in C3 0.505 0.763 0.196 0.278 0.531 N 960 984 988 987 982 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the quantity and quality of health supplies, medicine, and equipment in the health facility reported by health workers. Regression model 53 adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 41, below, provides an overview of reported satisfaction with the physical condition of health facilities. Health workers in the PBF arm were 31 percentage points more likely to be satisfied with the physical condition of the health facility building, relative to the pure control (p<0.01). While both the financing and the improved supervision arms had positive point values (10.6 percentage points and 9.6 percentage points, respectively), neither was statistically significant at the 10 percent cut-off, and workers in the full PBF arm were more likely to express satisfaction than were workers in either the additional financing (p<0.05) or the improved supervision (p<0.10) arms. Despite increased satisfaction with both the physical infrastructure and the quantity and quality of equipment, PBF did not have a statistically significant impact on health workers’ perceptions of their ability to provide high-quality care. The effects associated with the two partial treatment arms also failed to achieve statistical significance. Table 41: Physical condition of the health facility and ability to provide high quality care given health facility conditions† The physical condition of the health Your ability to provide high quality of facility building care given the current working conditions in the facility β se β se Post indicator -0.084 0.078 0.069 0.074 PBF/Post interact 0.306*** 0.111 -0.009 0.097 Control 1/Post interact 0.106 0.099 0.123 0.103 Control 2/Post interact 0.096 0.118 -0.129 0.116 p-value PBF vs. C1 0.036 0.184 p-value PBF vs. C2 0.074 0.287 p-value PBF vs. C3 0.006 0.926 Baseline mean in C3 0 .449 0.526 N 990 988 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the physical condition of the health facility and ability to provide high quality care given health facility conditions reported by health workers. Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. The interventions including financial support also appear to have positively impacted satisfaction with salary and benefits (Table 42). Health workers in the PBF arm were 9.1 percent points more likely to express satisfaction with their salary and 18.3 percentage points more likely to express satisfaction with their benefits (although only the latter was statistically significant (p<0.05)). Effects were even stronger in the additional financing group. Health workers at facilities receiving additional financing were 13.4 percentage points more likely to be satisfied with their salary (p< 0.10) and 28.7 percent points more likely to be satisfied with benefits (p<0.01). These effects are not seen in the improved supervision arm. While the point estimates associated with supervision are positive, they are relatively small and are not statistically significant. Satisfaction with living accommodations improved somewhat in all three groups, with point estimates suggesting that health workers were 10 to 16 percentage points more likely to express satisfaction after the pilot. However, these impacts are not statistically significant in any of the three groups. 54 Table 42: Health worker salary, benefits, and living accommodations† Your salary Your benefits (such as Living accommodations housing, travel allowance, bonus including performance bonus, etc.) β se β se β se Post indicator 0.036 0.036 -0.048 0.059 0.128 0.084 PBF/Post interact 0.091 0.061 0.183** 0.075 0.138 0.109 Control 1/Post interact 0.134* 0.069 0.287*** 0.083 0.096 0.109 Control 2/Post interact 0.053 0.061 0.037 0.098 0.159 0.117 p-value PBF vs. C1 0.572 0.17 0.675 p-value PBF vs. C2 0.587 0.113 0.845 p-value PBF vs. C3 0.138 0.016 0.207 Baseline mean in C3 0.054 0 .133 0.299 N 943 862 972 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on health worker salary, benefits, and living accommodations reported by health workers. Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. The point estimates of PBF’s impact on opportunities to discuss issues with supervisors, supervisors’ recognition of good work, opportunities to be rewarded for hard work, and opportunities for promotion were all negative (β=-0.08, -0.04, -0.02 and -0.11, respectively), although none were statistically significant. The estimated impact of the improved supervision arm was also generally negative and, with a point estimate of -25.1 percentage points, the impact of the improved supervision on opportunities for promotion was statistically significant (p<0.05). The estimated impact of the financing arm was positive, but was only statistically significant for the opportunity to be rewarded for hard work. The improvement in the financing arm was greater than the change in the in the PBF group for this outcome, and for opportunities for promotion. Table 43: Health worker relationships with their supervisor, opportunities to be rewarded for their work, and opportunities for promotion† Your opportunity Your immediate Your Your opportunities to discuss work supervisor's opportunity to for promotion issues with your recognition of be rewarded for immediate your good work hard work, supervisor financially or otherwise β se β se β se β se Post indicator 0.054 0.067 -0.008 0.077 0.066 0.063 0.137** 0.062 PBF/Post interact -0.081 0.095 -0.034 0.098 -0.019 0.107 -0.113 0.092 Control 1/Post interact 0.004 0.097 0.066 0.101 0.185* 0.102 0.085 0.087 Control 2/Post interact -0.061 0.103 -0.016 0.107 -0.113 0.115 -0.251** 0.100 p-value PBF vs. C1 0.386 0.249 0.096 0.033 p-value PBF vs. C2 0.848 0.841 0.477 0.189 p-value PBF vs. C3 0.396 0.729 0.861 0.222 Baseline mean in C3 0.663 0 .765 0 .302 0.152 N 980 975 971 918 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on health worker relationships with their supervisor, opportunities to be rewarded for their work, and opportunities for promotion. 55 Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. As shown in Table 44, the PBF intervention had a small but negative estimated impact on reported opportunities to upgrade skills through training and a small but positive estimated impact on reported opportunities to use skills on the job (neither measure met statistical significance). Point estimates for both measures were positive in the additional financing arm, and the financing appears to have positively impacted satisfaction with opportunities to use skills on the job (β =0.194, p< 0.10). The point estimates associated with the improved supervision arm to either use or upgrade skills did not meet the 10 percent cut-off for statistical significance. Satisfaction with the safety and security in the community and with available schooling for children were unchanged by the interventions. The point estimate in the PBF group was large at 13.9 percentage points, but did not reach statistical significance. The point estimates for the other two treat groups were small and non-significant. Health workers were also asked about their satisfaction with jobs, overall. While all three interventions had positive point estimates, none met the 10 percent cut-off for statistical significance. Relative to the pure control, the largest impact was seen in the PBF arm, with health workers 10.5 percent more likely to express satisfaction, followed by the supervision only group (5.3 percent more likely to express satisfaction), followed by the financing only group (4.8 percent more likely). Table 44: Health worker opportunities to upgrade their skills, to use their skills in the job, their safety and security in their community, the availability of schooling for their children, and their overall job satisfaction† Your The Safety and Available Overall, how opportunities to opportunities to security in the schooling for satisfied are you upgrade your use your skills in community your children with your job? skills and your job knowledge through training Post indicator 0.116 0.082 -0.013 0.072 0.093 0.083 -0.026 0.115 0.183** 0.085 PBF/Post interact -0.025 0.108 0.007 0.094 0.139 0.106 0.027 0.160 0.105 0.113 Control 1/Post interact 0.050 0.115 0.194* 0.103 -0.036 0.102 -0.006 0.171 0.048 0.113 Control 2/Post interact -0.075 0.121 0.007 0.118 -0.003 0.107 -0.044 0.172 0.053 0.114 p-value PBF vs. C1 0.497 0.047 0.045 0.841 0.608 p-value PBF vs. C2 0.666 0.999 0.124 0.672 0.639 p-value PBF vs. C3 0.819 0.937 0.189 0.866 0.355 Baseline mean in C3 0.309 0.694 0.619 0.347 0.337 N 967 989 984 726 986 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on health worker opportunities to upgrade their skills, to use their skills in the job, their safety and security in their community, the availability of schooling for their children, and their overall job satisfaction. Regression model adjusted for individual (age, sex, marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Health worker availability in the health facility 56 The facility survey asked the head of the health facility, or the most informed staff member, to list the names of the all of the health workers employed at the health facility. The staff roster also collected information on the post occupied by each health worker and on whether they were present on the day of data collection. Table 45 below presents results from analysis of the number of nurses present at the health facility on the day of data collection. There was a small and non-significant increase in the number of nurses present over the study period. The increase in the number of nurses in the PBF group was greater than in the full control group (p-value 0.010). Adding the coefficient on the interaction term of PBF and post to the coefficient on the post indicator (0.191+1.222=1.413) indicates that there was an average increase of almost 1.5 nurses present in PBF facilities over the study period. The coefficients on the two other treatment groups – additional financing and additional supervision – were not statistically significant; however, there was a larger increase in the full PBF group compared to the change in the additional supervision group. Table 45: Number of nurses present at the health facility on the day of the survey† Basic clinical equipment b se Post indicator 0.191 0.344 PBF/Post interact 1.222*** 0.468 Control 1/Post interact 0.738 0.475 Control 2/Post interact -0.172 0.475 p-value PBF vs. C1 0.291 p-value PBF vs. C2 0.003 p-value PBF vs. C3 0.010 Baseline mean in C3 2.725 N 366 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the number of nurses present at the health facility on the day of the survey. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Drugs and equipment in the health facility A composite indicator was created to assess any impact on the availability of basic clinical equipment (Table 46). The indicator included information on the presence of a clock, a child weighing scale, height measure, tape measure, adult weighing scale, blood pressure instrument, thermometer, stethoscope, fetoscope, otoscope, flashlight, stretcher, and wheelchair. Scores indicate the proportion of these thirteen pieces of equipment that was available at a given facility and range from 0 to 1. Point estimates indicate the estimated impact on this score. Both the PBF and the additional financing intervention arms resulted in large and statistically significant improvements in the availability of equipment. Facilities in the PBF arm had a 10.0 percentage point increase over that seen in the control (p<0.05), while those in the additional financing arm had an increase of 12.5 percentage points over the control (p < 0.01). This increase was not seen in the improved supervision arm; while the point estimate was positive, it was small and not statistically significant. There was no measurable difference in the impacts of the PBF and financing only arms, but there was statistically significant difference between the PBF intervention and improved supervision (p<0.05). 57 Table 46: Basic clinical equipment available at health facilities† Basic clinical equipment b se Post indicator 0.030 0.035 PBF/Post interact 0.100** 0.043 Control 1/Post interact 0.125*** 0.043 Control 2/Post interact 0.024 0.044 p-value PBF vs. C1 0.488 p-value PBF vs. C2 0.043 p-value PBF vs. C3 0.021 Baseline mean in C3 0.679 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the basic clinical equipment available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 47 shows the interventions’ impact on the availability of vaccination equipment, which includes a thermometer for the vaccine fridge, a cold box or vaccine carrier, a deep freezer, a refrigerator and ice packs. There is no evidence of a differential impact in any of the three arms. The point estimates are all very small, with large standard errors. Table 47: Vaccination equipment available at the health facility† Vaccination equipment b se Post indicator 0.102** 0.046 PBF/Post interact -0.013 0.063 Control 1/Post interact 0.021 0.060 Control 2/Post interact -0.037 0.070 p-value PBF vs. C1 0.563 p-value PBF vs. C2 0.724 p-value PBF vs. C3 0.842 Baseline mean in C3 0.702 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the vaccination equipment available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 48 shows the impact on delivery equipment, which includes the following: delivery bed, partograph, delivery light, aspirator, newborn resuscitation bag, newborn eye drops or ointment, scissors, umbilical cord clamp or sterile tape/tie, suturing material, examination gloves, sterile cotton gauze, hand soap or detergent, hand scrubbing brush, sterile tray, plastic container with plastic liner for the placenta, plastic container with a plastic liner for medical waste, adult stethoscope, Pinard or fetal stethoscope, blood pressure instrument, kidney basin, protective apron and plastic draw sheet, baby scale, needle holder, syringes and disposable needles, 16- or 18-guage needles, speculum, clamps, hand or foot operated suction pump, vacuum extractor, and a uterine curette. Both the PBF and the additional financing interventions had large and 58 positive impacts on the availability of delivery equipment. Scores in the PBF arm improved by 21 percentage points more than did those in the control and those in the additional financing arm increased by 18.9 percentage points relative to the control. While there was a positive point estimate on the impact of the improved supervision intervention group, the difference (estimated at 8.2 percentage points) does not meet the cut-off for statistical significance. The impacts of the PBF and financing arms are not statistically significantly different from one another. However, the impact of PBF is significantly larger than that seen in the improved supervision arm (p<0.10). Table 48: Delivery equipment available at the health facility† Delivery equipment b se Post indicator 0.016 0.047 PBF/Post interact 0.209*** 0.064 Control 1/Post interact 0.189*** 0.06 Control 2/Post interact 0.082 0.07 p-value PBF vs. C1 0.729 p-value PBF vs. C2 0.061 p-value PBF vs. C3 0.001 Baseline mean in C3 0.534 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the delivery equipment available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 49 shows the impacts on an index of general medicines, including paracetamol, amoxicillin tabs or syrup, ORS, iron tabs, and cotrimoxazole. While all the point estimates for the intervention arms are positive, they are small relative to their standard errors and no there is no evidence of an impact, relative to either the pure control or one of the other intervention groups. Table 49: General medicines available at the health facility† General medicines b se Post indicator 0.045 0.039 PBF/Post interact 0.064 0.058 Control 1/Post interact 0.089 0.063 Control 2/Post interact 0.050 0.064 p-value PBF vs. C1 0.701 p-value PBF vs. C2 0.830 p-value PBF vs. C3 0.270 Baseline mean in C3 0.768 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on general medicines available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. The same is true for family planning methods, shown in Table 50. The index is comprised of condoms, oral contraceptive tablets, Depot Medroxyprogesterone Acetate (DMPA), and implants. The estimated impact for PBF, in particular, is large at 16.8 percentage points and 59 statistically significant (p < 0.10). The point estimates indicating the effect of the other two different treatment arms are also positive, though they are not large enough to pass the test of statistical significance. Also, the effect in the PBF group was statistically significantly different from the effects observed in the other treatment groups. Table 50: Family planning methods available at the health facility† Family planning b se Post indicator -0.105* 0.063 PBF/Post interact 0.168* 0.091 Control 1/Post interact 0.078 0.097 Control 2/Post interact 0.100 0.097 p-value PBF vs. C1 0.361 p-value PBF vs. C2 0.491 p-value PBF vs. C3 0.067 Baseline mean in C3 0.482 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on family planning methods available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 51 assesses the impact of the interventions on the availability of malaria medicines, including Coartem, ACT lumefantrine artesunate tablet, ACT lumefantrine artesunate syrup, and sulphadoxine-pyrimethamine. There is no evidence of any effect on malaria treatment, as indicated by the very small point estimates and large standard errors. Table 51: Malaria treatment medicines available at the health facility† Malaria medicines b se Post indicator -0.048 0.063 PBF/Post interact -0.014 0.080 Control 1/Post interact -0.028 0.083 Control 2/Post interact 0.021 0.093 p-value PBF vs. C1 0.844 p-value PBF vs. C2 0.672 p-value PBF vs. C3 0.864 Baseline mean in C3 0 .646 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on malaria treatment medicines available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. Table 52 provides an overview of the estimated impact on the availability of vaccines, including Bacille Calmette-Guerin (BCG), Oral Polio Vaccine (OPV), tetanus toxoid, Diptheria Tetanus and Pertussis (DTP), Hepatitis B (HBV), measles, Hemophilus influenza B (Hib), DPT, Hepatitis and Hemophilus influenza (Pentavalent). While the point estimates for both the PBF and the financing arms were positive, indicating an approximately 13 and 11-percentage point increase respectively over the control, neither of these met the 10 percent cut-off for statistical 60 significance. While still positive, the point value for the improved supervision arm was smaller, at 5.3 percentage points, and also failed to meet the cut-off for statistical significance. Table 52: Vaccines available at the health facility† Vaccines b se Post indicator -0.116** 0.057 PBF/Post interact 0.131 0.088 Control 1/Post interact 0.113 0.091 Control 2/Post interact 0.053 0.093 p-value PBF vs. C1 0.850 p-value PBF vs. C2 0.432 p-value PBF vs. C3 0.136 Baseline mean in C3 0.530 N 370 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on vaccines available at the health facility. Regression model adjusted for facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. The quality of consultations for children under 5 years of age Enumerators observed a total of 575 child health consultations. For each, enumerators compared the exchange against a standardized checklist and noted whether the health worker performed the following eleven routine activities: greeted the patient, washed hands, asked age, duration of the complaint, if the child is able to drink or breastfeed, if the child vomits everything, if the child is lethargic, if the child took any medicine and if the child had diarrhea. An overall quality score was calculated for each visit by calculating the proportion of these activities that was conducted. As shown in Table 53, there is no evidence of impact in any of the three intervention groups. Both the PBF and the improved supervision groups have small but negative point estimates with large standard errors and, while the additional financing group has a positive estimated impact (4 percentage points), this too fails to meet the cut-off for statistical significance. Table 53: Overall quality score of child health consultations for children under 5 years of age β se Post indicator 0.030 0.041 PBF/Post interact -0.021 0.055 Control 1/Post interact 0.044 0.056 Control 2/Post interact -0.029 0.058 p-value PBF vs. C1 0.230 p-value PBF vs. C2 0.888 p-value PBF vs. C3 0.663 Baseline mean in C3 0 .511 N 575 * = p < 0.10, ** p < 0.05, *** p< 0.01 61 † Results from difference-in-differences regression models examining the effect of PBF on the overall quality score from child health consultations from direct observation of child health consultations. Regression model adjusted for individual child-level variables (age, sex), maternal variables (marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). Standard errors were clustered at the health facility level. The quality of antenatal care Enumerators observed 725 ANC consultations. They compared each exchange against a standardized checklist and noted whether the health worker performed the following eleven routine activities: took a background,2 asked about past issues,3 asked about current issues,4 provided iron supplementation, gave advice about warning signs,5 helped to prepare for the birth,6 checked HIV status, tested for syphilis, provided malaria prophylaxis, discussed appropriate nutrition, and checked the following vital signs: blood pressure, weight, conjunctiva, hemoglobin, rhesus, urine glucose, uterine size, fetal heartbeat and fetal presentation. As with the child health consultations, these data were used to calculate aggregate quality scores. National protocols provide guidelines for care that is specific to the number of prior visits and gestational age of the pregnant women. As not all activities are appropriate for all consultation, the quality indices were adjusted to reflect variations by gestational age (<32 weeks, 32 to 35 weeks, and >35 weeks) and whether or not the patient was experiencing her first pregnancy. Results are presented in Table 54. We see a strong positive trend in the quality of ANC over time. On average, consultation scores improved by improved by 12.9 percentage points between baseline and endline (p<0.05). As shown in Table 54, there were no differences in any of the treatment groups in the change in ANC quality relative to the full control. Table 54: Overall quality score of ANC consultations† 2 A composite score ranging from 0-1 indicating whether the worker asked about the patient’s age, medicines, date of last menstruation 3 A composite score ranging from 0-1 indicating whether the worker asked whether the patient had any prior deliveries, stillbirths, neonatal deaths, abortions, heavy bleeding during or after delivery, or assisted delivery. 4 A composite score ranging from 0-1 indicating whether the worker asked whether the patient had any bleeding, fever, headache or blurred vision, swollen face or hands, tiredness or breathlessness, felt the baby move, or if the client noticed any other symptoms or problems related to the pregnancy. 5 A composite score ranging from 0-1 indicating whether the worker warned the patient to watch for vaginal bleeding, fever, excessive tiredness or breathlessness, swollen hands and face, and severe headache or swollen vision. 6 A composite score ranging from 0-1 indicating whether the worker advised the client to prepare for the birth, including arranging money and transportation, advised to have skilled assistance at delivery, discussed what items to have on hand, emphasized the importance of immunization and the importance of exclusive breastfeeding. 62 ANC Quality β se Post indicator 0.129*** 0.029 PBF/Post interact -0.056 0.042 Control 1/Post interact 0.015 0.045 Control 2/Post interact -0.042 0.042 p-value PBF vs. C1 0.131 p-value PBF vs. C2 0.742 p-value PBF vs. C3 0.191 Baseline mean in C3 0.592 N 729 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on the overall quality score from child health consultations from direct observation of child health consultations. Regression model adjusted for individual child-level variables (age, sex), maternal variables (marital status, education level) and facility-level control variables (type of health facility public/private/religious, urban/rural). In addition to the standard controls, also controlled for whether it is the first pregnancy (Y/N) and where in the pregnancy the woman is (<32 weeks, 32-35 weeks, >35 weeks). Standard errors were clustered at the health facility level. Discussion 63 Health sector context Despite being one of the wealthier countries in the Central African region, and the country’s relatively high health spending of $59 per capita in 2014 (The World Bank 2016b), Cameroon’s health indicators resemble countries that spend much less on health care (ICF International 2012). Although the health sector budget has more than doubled in recent years, the lion’s share of resources has been allocated for administration, rather than to the front lines where health services are provided. This has resulted in a scarcity of funds to meet operating expenses incurred in the day-to-day business of a district health system (e.g., consumables, drugs, regular maintenance, community outreach, etc.). In order to more directly link payments and funding at the provider level with the quantity and quality of health services performed, after an initial pilot in the Littoral region, Cameroon started a PBF pilot in 2012 in 14 districts in three regions of the country: East, South East and South West. Performance-based financing is a health systems management tool designed to increase the efficiency of health system inputs to improve the coverage and quality of priority maternal and child health services by paying health facilities bonuses linked to the quantity and quality of services delivered. Impact evaluation design and research questions The PBF pilot was accompanied by a prospective randomized impact evaluation. In order to distinguish the influence of the different components of the PBF reform, the evaluation compared four arms: (1) the standard PBF package, (2) the same level of financing but not linked to performance, and with the same levels of supervision, monitoring, and autonomy as PBF, (3) no additional resources or autonomy, but the same levels of supervision and monitoring as PBF, and (4) pure comparison. The randomization took place at the health facility level and overall the four study groups were well balanced at baseline. The analysis also established that the levels of financing between the first two groups (standard PBF and additional financing) was indeed equivalent at endline. The evaluation used a combination of household and health facility surveys conducted at baseline and endline to assess the impact of the interventions. The impact evaluation focused on the following principal research questions: 1. Does the PBF program increase the coverage of MCH services? 2. Does the PBF program increase the quality of MCH services delivered? 3. Is it the enhanced monitoring & evaluation and supervision or the link between payments and results that leads to improvements observed in quality or coverage? 4. What is the contribution of enhanced supervision and monitoring to improving MCH service coverage and quality in the absence of increased autonomy or additional financial resources? The discussion is structured around these main research questions, starting the coverage of MCH services, followed by the quality of those services. We then regroup questions 3 and 4 above into a discussion of the differences in the measured impacts of the three intervention arms. We end by discussing the study limitations. 64 MCH Coverage indicators For the PBF arm (arm 1), results from the health facility survey analysis show significant increases in coverage of the childhood vaccinations (including the polio 3 vaccine) and maternal immunization against tetanus as well as the coverage of modern methods of family planning. In the household survey analyses, child vaccination was the only indicator with a statistically significant increase in the PBF group (17 percentage points, p=0.076), but these results at the household level should be interpreted with caution due to the high degree of facility bypassing observed in the data with close to half of households seeking care in a non-assigned health facility. The results from the household survey analysis are however broadly consistent with the results from the health facility survey analysis which are not affected by the measurement error introduced by the health care shopping behavior of households: indeed both analyses found no significant changes for timely ANC and in-facility deliveries, while both analyses found that PBF had a positive impact on child vaccinations. The absence of impacts for indicators that were seen as priorities, such as ANC and in-facility deliveries is surprising. We hope that the qualitative study will help us better understand why these indicators did not move as anticipated. Potential explanations might be that the coverage for these indicators was already relatively high at baseline in the IE districts (77% for facility- based deliveries, 88% for ANC2) or that the Ministry of Health introduced the obstetrical kit intervention in all facilities at the same time as PBF. It is also possible that the supply-side incentive for providers was not sufficient given existing user fees, which might act as a barrier on the demand-side. A discussion about combining demand-side and supply-side incentives would then be useful, especially in light of the fact that an indicator such as skilled deliveries has not increased more than 2% in Cameroon for the past 20 years. Out of pocket payments Out-of-pocket health expenditures decreased for households in the PBF arm, in particular for unofficial provider fees (-2254 CFA or $3.64, p=0.086) and laboratory and x-rays fees declined by (-1473.44 CFA or $2.38, p=0.06). While these effects are only measurable from the household survey data and not from the exit interviews conducted as part of the health facility survey, we believe that the information about out-of-pocket payments is more reliable in the household survey than from exit interviews because respondents are generally more comfortable discussing topics like unofficial payments in the privacy of their households. Patient Satisfaction The results indicate a significant increase in overall satisfaction expressed during exit interviews among parents/care takers of children less than 5 who received child health consultations at health facilities in the PBF arm. Similarly, there was an 8.6 percentage point increase in overall satisfaction in the PBF group among women answering exit interviews after receiving antenatal care is not statistically significant at conventional thresholds (p<0.1). Direct observation The health facility surveys included direct observations of ANC and child health consultations. The analysis of those observations doesn’t show evidence of significant changes in the interpersonal quality or the technical quality of ANC and child health consultations across the different study groups. 65 Equipment and Staffing In terms of structural quality, the PBF arm (T1) as well as the additional financing arm (C1) saw increases in the average availability of necessary equipment, particularly materials for delivery and neonatal care. Health worker motivation The availability of human resources for health also appears to have improved in the PBF arm, with more qualified health workers present on site than in the other arms as well as an increase in their financial and non-financial job satisfaction. In particular, health worker satisfaction with the availability of equipment, medicines, consumables and infrastructure increased. It is important to note that the same trends were observed the group receiving additional financing (C1). Comparison across intervention arms Overall, the impact evaluation results reveal significant increases in the PBF arm for several indicators (child and mother vaccinations, use of modern family planning), but not for other such as antenatal care visits and in-facility deliveries. Structural quality as measured by equipment availability, staff presence and staff satisfaction, improved in the PBF group. However, despite an increase in providers and supplies available at health facilities, PBF did not increase the completeness of service provision during antenatal care and child health consultations. Importantly, out-of-pocket health expenditures decreased for household in the PBF arm, including unofficial payments and this decrease in revenue did not come at the cost of process quality: there were no negative spillover effects on completeness of services and advice provided during antenatal visits and consultations for children under 5. Perhaps not surprisingly, then, given decreased out of pocket costs, and improvements in structural and process quality, client satisfaction also increased for medical consultations for children younger than 5. While some –but not all -of the improvements measured for PBF were also observed in the additional financing arm C1, few improvements were observed in the group C2 offering enhanced supervision without additional financing or financial incentives. The comparison between the PBF and the C1 group is delicate because the two interventions share many similarities: same supervision and monitoring mechanisms, same level of managerial autonomy and increased financing. The only difference was that in the PBF group (T1) the additional financing was linked to the performance of the individual facility while in the C1 group it was linked to the average performance of the PBF facilities in the same district. It is not impossible that this distinction might not have been salient enough among the health facility management and staff, explaining overall similar results. The lower impacts obtained in the C2 group however suggest that reinforced supervision is not sufficient to change behaviors and improve outcomes. Additional financing appears to be required and its impact seems in some instances stronger when linked to results as in PBF. Study limitations This study has several limitations that we have tried to acknowledge in this report. The randomization for this study was at the health facility level. This is beneficial from the point of view of statistical power. From an operational and public health perspective, however, 66 randomizing at the district level would have make more sense given the proximity of some facilities. Indeed, the risk with facility-level randomization is that neighboring facilities from different groups might learn from each other and apply principles outside their treatment group. However, this was not feasible given that the Government of Cameroon had already decided and announced which districts would be included in the PBF pilot. Randomization at the district level was therefore not an option. We have analyzed in detail the phenomenon of health care bypassing behavior whereby households look for health care beyond the closest health facility. We found that health care shopping behavior by households was widespread in Cameroon at baseline in 2012 and continues to be widespread at endline in 2015, but does not appear to be a consequence of the introduction of PBF. Overall, the results do not suggest that the health care seeking behavior is driven or even significantly influenced by the introduction of PBF or the other interventions in C1 and C2 limiting the concerns for systematic bias. However, this bypassing behavior likely leads to estimates which are below the true causal effect of the intervention. This is a substantial limitation of the household survey analysis that needs to be kept in mind. While overall we found that the results from the household survey and the health facility survey analyses were consistent, for example on the absence of impacts for the ANC and skilled delivery indicators and the presence of positive impact on immunizations, we also noted some discrepancies. In some cases, because of the nature of the information collected, one of the two data collection method might be superior. For example, patients might be more open about reporting unofficial payments in a household survey than during an exit interview conducted within or in the vicinity of the facility. In contrast, women might be reluctant to report family planning utilization at home and therefore facility-level data for this indicator might be more reliable. Another potential limitation is that the differences between the three intervention study groups were sometimes subtle. This was certainly the case, as discussed above, between the PBF group and the C1 group that offered all the elements of PBF except the direct link between individual facility performance and additional financing. It is not obvious that all these differences in intervention design have been fully grasped by staff and management. We should also acknowledge that the monitoring of adherence to national guidelines done as part of the monitoring and supervision intervention in T1, C1 and C2 facilities was not ideal from an evaluation point of view because it means that all three treatment groups receive a separate intervention which the control group does not receive. This is obviously not something that could have easily been avoided from an implementation perspective, and it seems likely that the impact of these protocols is small. Finally, this report relies on quantitative household and health facility surveys. A companion qualitative study has been conducted and its analysis is ongoing. The qualitative analysis will help understand and interpret some of the impacts measured – or their absence – and will shed light on possible mechanisms. 67 Conclusions and Policy Recommendations From a policy point of view, these impact evaluation results suggest the following lessons. 1) In general, PBF is an efficient mechanism to bring payments and funding at the provider level, leading to significant increases in coverage (child and maternal immunization, family planning, HIV testing) and improvements in structural quality of care. It also leads to a decrease in out-of-pocket payments, in particular unofficial payments. 2) For many of those outcomes, the differences between the PBF group (T1) and the additional financing group (C1) are not significant. It should be noted that the C1 group offered all the elements of PBF except the direct link between individual facility performance and additional financing. It is not obvious these differences in intervention design have been salient enough for staff and management. 3) There was, however, a clear effect for the importance of additional financing plus reinforced supervision through PBF instruments (comparing groups T1 and C1 vs groups C2 and C3). Enhanced supervision and monitoring are not sufficient to improve MCH outcomes. 4) The absence of impacts for some MCH indicators such as skilled deliveries and ANC visits was surprising. It is possible that the supply-side incentives for providers were not sufficient given existing user fees which might act as a barrier on the demand-side. A policy discussion about combining demand-side and supply-side incentives would be useful. 5) In terms of quality of care, most of the positive impacts were observed on structural quality. However, despite an increase in providers and supplies available at health facilities, PBF did not increase the completeness of service provision (content of care) during antenatal care and child health consultations. Further reflection and efforts should be devoted to identify mechanisms to incentivize or otherwise improve the content of care beyond equipment, supplies and staff availability. 68 Appendix A Intervention group comparison table T1 C1 C2 C3 Complete PBF with performance PBF with subsidies that are Only supervision without Status quo bonuses for medical personnel not linked to performance bonuses or autonomy Contract Classic PBF contract Contract stipulating the Contract stipulating technical No contract conditions of PBF for support in the form of verification and supervision supervision Business plan Yes Yes Simple business plan focused No business plan on intensified supervision Quality evaluation Quality evaluation and feedback Quality evaluation with Quality evaluation with Quality evaluation with with quality taken into account in feedback as in T1, but no effect feedback as in T1 written feedback twice a bonus payment on payment year Review/verification Review and verification of service Review and verification of Review and verification of Single quarterly of service amounts quantities service quantities service quantities statement without verification of the quantity of services produced Payment Payments tied to performance Payments not tied to No payment No payment performance Management autonomy avec Management autonomy avec No management autonomy, No management Management retenue des toutes les recettes retenue des toutes les recettes continuation the status quo autonomy, continuation autonomy system the status quo system Monthly activity report submitted Yes Yes Yes Yes to district 69 Appendix B Regional health facility maps 70 71 72 Appendix C PBF subsidy table Output Indicators for the Minimum Package of Health – Health Center N° Curative Care Definition Support documents for data Unit cost Monthly Target collection in FCFA Calculation (Basic Subsidy) 1 Out Patient Consultations Number of persons consulting the health Outpatient consultation register 200 Total population of (new cases): Nurse center with a new episode of illness or register used for curative catchment area /12 x 80% (consulted by nurses) care consultations 2 Out Patient Consultations Number of persons consulting the health Outpatient consultation register 650 Total population of (new cases): Doctor center with a new episode of illness or register used for curative catchment area /12 x 20% (consulted by Medical Doctors) care consultations 3 Out Patient Consultations Number of persons consulting the health Outpatient consultation register 1000 Total population of of epidemics (new cases): center with a new epidemic case (consulted or register used for curative catchment area /12 x 20% Doctor or nurse (free) by Medical Doctors or nurses) care consultations or special epidemics registers 4 Hospital bed days Total Number of days spent by all the Inpatient (hospitalization 400 Total population of (observation/Hospitalizati inpatients in the health center (for register of the health facility catchment area /12 x 30% on) observation or awaiting referral) period limited to a maximum of 48 hours 5 Hospital bed days Total Number of days spent by all the Inpatient (hospitalization 1500 Total population of (observation/Hospitalizati inpatients epidemic cases in the health center register of the health facility catchment area /12 x 30% on) for epidemic cases (for observation or awaiting referral) period x20% (free) limited to a maximum of 48 hours 6 Minor surgery cases Total number of New cases of minor surgery Minor surgery register 1500 Total population of treated in the health facility (incision of catchment area /12 x 7% abscesses, wound sutures, circumcisions etc.) 73 7 Referral received in the Total number of referred patients who are Referral register of the health 1500 Outpatient consultations hospital received at the referral hospital center, referral forms at the x 0.5% level of the Hospital, consultation registers of the hospital, Hospitalization registers Preventive Services/Care 8 Children Completely Children 0-11 months who received all of Vaccination register of the 2500 Total population of Vaccinated the following vaccines (BCG, Pentavalent 1, health facility catchment area x 4% /12 Pentavalent 2, Pentavalent 3 yellow fever x 100% and measles) 9 VAT2 or VAT3 or VAT4 Total number of women who received either ANC Register and/or VAT 1500 Total population of or VAT 5 VAT2 or VAT3 or VAT4 or VAT5 vaccination register catchment area x 4.5% /12 x 100% 10 Home visits Number of homes visited which had : Home visits register signed by 2500 To be defined appropriate collection and disposal of the Health committee household refuge; a latrine in good state ; representatives and the village appropriate use of mosquito bed nets and use chiefs or quarter heads of portable water. 11 Vitamin A Number of children 6 to 59 months who Vit A supplementation register, 20 Total population of supplementation received Vit A Vaccination Register catchment area x 16.5% (distribution) /12 x 100% x 3 12 HIV positive Pregnant Number of HIV positive Pregnant Women PMTCT Register 7000 Total population of Women put on ARV put on ARV prophylactic treatment catchment area x 4.5% x prophylactic treatment according to the national PMTCT protocol in 4% /12 the month 13 Newborn management of Number of babies born of HIV positive PMTCT Register 7000 Total population of a baby born of an HIV mothers who are placed on PMTCT protocol catchment area x 4% x positive mother. in the month according to National directive 4% /12 14 Voluntary Counseling and Number of people who came to the health VCT Register 1000 Total population of Testing for HIV/AIDS facility for HIV/AIDS voluntary counseling catchment area x (15-59) and testing and who collected their results /12 x 90 % 15 Cases of STIs treated Number of new cases of STIs diagnosed and Outpatient consultation register 400 Total population of correctly treated in the month according to catchment area x 5% /12 national protocols (Syndromic approach) x 90 % 74 16 Cases of TB diagnosed Number of new cases diagnosed positive by TB and Lab registers 10,000 Total population of positive by Microscopy Microscopy in the health facility catchment area x (87/100,000) /12 x 30% 17 Cases of TB treated and Total number of positive TB cases on TB register, Lab register 20,000 Total population of healed treatment who were completely healed in the catchment area x month (87/100,000) x 85% /12 Reproductive Health 18 Normal Assisted Delivery Total number of normal deliveries carried Deliveries Register (Maternity 2500 Total population of out by qualified (or skilled) staff (nurses) in Register) catchment area x 4.5% the facility in the month /12 x 90% x 80% 19 FP : New or old acceptants Total number of both old and new acceptants Family Planning Register 1200 Total population of on oral pills of injections of family planning who are currently on oral catchment area x 22% /12 pills or injections x 21% x 4 20 FP : Implants and IUD Number of new cases of Implants and/or Family planning register 3000 Total population of IUD carried out in the month catchment area x 22% /12 x 2% Total number of new cases of curettage (post Maternity and theater register 21 Post abortive Curettage 3500 Total population of abortive) carried out in the facility in the (spontaneous or induced) catchment area x 5% x month 10% x 50% /12 22 ANC1 or ANC2 or ANC3 Total number of pregnant women who ANC Register 500 Total population of or ANC4 consulted the health facility in the month catchment area x4.5% /12 either for ANC1 or ANC2 or ANC3 or x 3 x 75% ANC4 23 IPT1 or IPT2 or IPT3 Total number of pregnant women who ANC Register 500 Total population of consulted the ANC service of the facility in catchment area x4.5% /12 the month and who took either IPT1 or IPT2 x 3 x 75% or IPT3 75 Appendix D PBF quality evaluation Protocol Protocol 1. GENERAL INDICATORS respected not respected 1. The Health area map available and pasted up: Map pasted up in the facility showing the other health structures, villages/quarters, major roads, 1 0 natural obstacles, major bridges and distances 2. The Monthly activity reports (NHMIS), Business plans and other action plans, minutes of meetings and other important services documents (personnel files and administrative letters) well arranged. Documents in files and arranged on shelves and available to the person in charge at the moment (chief 2 0 of center, assistant chief of center, nurse on call etc.) 3. Working Hours and calls roster available and pasted up. 1 0 4. A 2-3 year health development plan of the facility available. 1 0 5. Minutes of Technical meetings of the health facility done on a monthly basis and available. Verify that of the previous month which should contain: date, starting time and time of end of meeting, agenda of 1 0 the meeting, signed attendance sheet, points of discussion and signature of the chair person and the secretary. 6. Existence of uncompleted referral slips (at least 10) 1 0 7. Availability of a telephone set or radio for communication between the health facility and the first referral health facility. Radio or telephone set belonging to the health facility and in good functional 1 0 state with batteries and airtime (if cell phone) of not less than 1000frs 8. A kitchen available for the hospitalized patients and in good state : Wall made of bricks or iron/aluminum sheets, roofed with iron/aluminum sheets or tiles, a dustbin for kitchen waste, a water 1 0 tap or source at less than 10m or at least 50l of water available 9. Existence of a mortuary A room or a small structure (building) 1 0 Total Points - 10 points maximum ….. / 10 XXXXXX 76 Protocol Protocol 2. SEMESTER BUSINESS PLAN respected not respected 1. Existence of a health facility semester Business Plan. The evaluator verifies the current business plan. Different strategies: outreach (EPI, surveillance, 4 0 FP, ANC, PMTCT, Home visits, LLITN etc.) 2. Business Plan developed in collaboration with the major stakeholders - Different heads of services of the facility, Health committee and Management committee chair 3 0 persons (if possible) The minutes of the planning meeting with attached attendance sheet. 3. Monthly business plan analysis (this can actually be an item on the agenda of the monthly evaluation and planning meeting of the health facility). 3 0 ….. / !B15 Is Not Points TOTAUX - 10 points maximum XXXXXX In Table Protocol respected Protocol 3. HYGIENE & STERILISATION not respected 1. Health facility has a fence in good condition If a planted hedge, it should be well streamed. If made of sticks or bricks, there should be no passage 2 0 for animals or persons. 2. Dustbin available on the yard Dustbin with a cover and not full. 1 0 3. Presence of Latrines in good state and adequate number - At least 3 latrines (one for male patients, one for female patients and one for staff) - The floor without cracks and has just one squatting hole which is well covered. 2 0 - Doors closed without flies - Toilets constructed with bricks and roofed with iron/aluminum sheets or tiles or straw. - Recently cleaned without excreta visible 4. Bathrooms available and in good state and number - At least 3 bathrooms; 2 0 - Bathroom with flowing water or container with at least 20 l of water - Waste waters draining into a suck-away. 5. Existence of an incinerator in good state and a placenta pit - Functional incinerator and emptied ; 5 0 - Placenta pit with a cover ; 77 6. Waste Pit for Non-organic (non septic waste) available Pit of at least 3m deep and covered, without septic waste or non-biodegradable material. 1 0 7. Cleanliness of the yard. Yard not littered with dirt or dangerous objects (needles, gloves, used gauze) 2 0 8. Maintenance of the yard 1 0 Grass cut, – Laune well kept – non animal droppings 9. The staff sterilizes instruments according to norms and standards 3 0 Sterilizing machine in good working condition and operational instructions pasted by it. 10. Hygienic conditions well respected in the treatment room, the dressing room, the injection room. Dustbin with cover for septic waste available – Safety box for needles and sharps available ad used 1 0 TOTAL POINTS - 20 points maximum …. / 20 XXXXXX 4. FINANCIAL SECTION: INCOME, RUNNING COST EXPENDITURE, PERFORMANCE Protocol Protocol INCENTIVES, ALLOCATIVE INDEX. respected not respected 1. Financial and accounting documents available and well kept. - Monthly financial report available and correctly filled-in (up to date without cancelations and 4 0 printing) - Theoretical Balance of the cash book corresponds to the liquid cash in the safe. 2. Estimated monthly expenditure approved by the health facility chief and the management committee chair person 2 0 2. The basic incentives plus the performance incentives does not exceed 50% of the PBF subsidies of the health facility. The evaluator verifies the sum of the incentives and compares with the total income of the health 2 0 facility 4. There is an established mechanism of calculating staff incentives and known to all: The criteria are based on: -(a) Basic performance index + -(b) Longevity + 2 0 -(c) responsibility post + -(d) over-time / absence + (e) Quarterly staff performance evaluation 78 TOTAL POINTS - 10 points maximum ….. / 10 XXXXXX Protocol Protocol 5. OUTPATIENT CONSULTATIONS /EMERGENCIES respected not respected 1. Good waiting conditions for OPD 1 0 With enough benches and or chairs ; shaded from rain and sun 2. Price list for user charges pasted up. Les tarifs du recouvrement de coût sont affichés 1 0 List conspicuously visible to all patients before consultation 3. Presence of a triage system with numbered cards to follow a cue 1 0 4. OP consultation room in good condition Wall constructed with bricks, plastered and painted. Floor cemented without cracks, ceiling in good condition and well sealed, window shutters made of wood or glass and window blinds mounted, Doors 2 0 in good functional state with good locks 5. OP consultation room and waiting room separated to ensure confidentiality during consultation. 1 0 Room with closed doors and windows with blinds – No direct passage 6. OP consultation room and/or emergency room lighted at night. Electricity or Solar energy or Lamp 1 0 with enough kerosene reserve 7. All the OPD consultations are carried out by a qualified nurse (At least a State Registered Nurse) Verify the qualification of the consulting nurse by an exit interview of the patients on the day of the 2 0 quality assessment. 8. The staff is in appropriate dressing Doctors : White long sleeve lab coats Nurses : Shirt-sleeve white lab coat 1 0 - Buttoned with an identification tag (batch), covered shoes, no shorts, nails well trimmed without nail cortex non hanging jewelry. 9. Proper and correct numbering in the OPD Register. Correct numbering and closed at the end of the month 1 0 10. OPD services available 7 days on 7 The evaluator verifies the entries of the OPD registers for the last 3 Sundays. 1 0 11. Malaria case management protocol pasted on the wall in the OPD. Simple and severe malaria case management National protocols 3 0 12. Correct management of simple malaria. Check in the OPD register for the 5 cases of simple malaria treated (AS/AQ ; A/L) 1 0 79 13. Correct management of severe malaria. Check in the OPD register for the 2 cases of severe malaria treated (antipyretics, anticonvulsives in 1 0 the case of convulsions and immediate referral) 14. Correct management of Acute Respiratory Infections (ARI). 2 0 The Ordinogramme available and applied. 15. Correct management of diarrhea 2 0 The Ordinogramme available and applied 16. The proportion of patients treated with antibiotics < 50% Check out in the OPD register for the last 30 cases consulted to make sure that those who received 2 0 antibiotics are less than 14. 17. Diagnosis for TB in chronic cough patients. Check in the OPD register and verify if the last 5 cases of cough (more than 2 weeks) were requested a sputum test. 1 0 18. Trend curves (graphs) drawn and pasted up and up-to-date for: contraceptive (FP) use, vaccination, rate of use of the facility 1 0 19. Blood pressure machines (sphygmomanometers) and stethoscope available and in good functional state. 1 0 Take the PB of somebody to verify the functional state. 20. Availability of a thermometer in good state 1 0 21. Availability of an otoscope in a good functional state. Inspect for : charged batteries and good lighting 1 0 22. Existence of a Salter baby scale in good functional condition 1 0 Scale regulated at “zero� point and carrier bag not torn and clean 23. An examination bed available (metallic or wood) with a mattress 1 0 24. Existence of a Adult weighing scale in good condition Evaluation takes his/her weight and verifies that gauge returns to « zero ». 1 0 25. Availability of tongue depressors. 1 0 26. Existence of a height measuring scale on good condition and figures visible 1 0 27. Existence of Weight/height comparison tables (chart) 1 0 80 28 .Evaluation of the nutritional status of all the children <5 years consulting in the facility 3 0 28. Evaluation of the nutritional status of the mothers whose sick children are below 6 months 2 0 29. A Nutritional status assessment register available and up to date 2 0 31. Management of Malnutrition according to National protocol 3 0 TOTAL POINTS - 44 points maximum …. / 44 XXXXX Protocol Protocol 6. FAMILY PLANNING respected not respected 1. All the FP consultations carried out by at least a Brevete Nurse 2 0 Verify that the FP consultation forms are completed and signed by a Brevete Nurse 2. FP consultation space available and ensures confidentiality. Room with closed doors and windows with blinds – No direct passage 2 0 3. Wall chart or Pictorial Booklet or Flip chart available showing demonstrations on the different modern methods of FP 1 0 4. Health facility reaches at least 90% of the planned quarterly target for oral and injectable contraceptives. 4 0 5. Security Stock for oral and injectable contraceptives available 10.000 inhabitants = 147 doses DEPO and pilles 3 plaquettes / 4 = 36 doses 3 0 6. IUD Methods available and staff capable of placing them. - At least 5 IUD available 2 0 7. Norplan available and staff capable of placing them - At least 5 available 2 0 8. Strategy in place to refer couples for Tubal ligation and/or vasectomy - Good referral system elaborated - Mechanisms for outreached developed by the reference hospital in collaboration with the health 2 0 center/CMA in place (collaboration document in exists) 9. The FP register in available in well filled-in (up to date and all the rubrics completed) 3 0 81 10. FP forms are available and well filled-in (verify 5 forms) : BP, weight, varicose veins 3 0 TOTAL POINTS – 24 points maximum …… / 24 XXXXXX Protocol Protocol 7. LABORATORY respected not respected 1. An Assistant Lab Technician available (at least) 1 0 2. The laboratory of open 7 days on 7 and operational The evaluator verifies the last 2 Sundays in the Lab register. 1 0 3. The lab results are correctly recorded in the lab register and conform with the results in the patient booklet or lab request slip 1 0 The evaluator verifies the last 5 results 4. The possible lab tests pasted up 0.5 0 5. Demonstrations of the different forms of parasites available: - As plastified papers or booklets in color or pasted up. 0.5 0 - Thick blood film: Vivax, Ovale, Falciparum, Malariae - Stool : Ascaris, entamoebae, ankylostome, schistosome 6. Functional microscope available – Lenses functional –immersion oil – mirror or electricity – slides – slide covers – GIEMSA available 2 0 7. A functional centrifuge machine available 1 0 8. –Environment : clean working surfaces (staining tables) - equipment and material well arrange - Waste disposed of: 1 0 - Organic waste in covered dust-bin - Security boxes available and filled boxes destroyed in an incinerator 9. The lab staff steeps used slides and pipettes in an antiseptic solution (instructions or protocol pasted up) 1 0 10. At least 10 pregnancy tests in stock in the lab. 1 0 TOTAL POINTS - 10 points maximum …… / 10 XXXXXX 82 Protocol Protocol 8. HOSPITALISATION /OBSERVATION WARD respected not respected 1. Equipments available and in good condition Beds with mattresses covered with ‘Macintouch (plastic) not torn, LLITN, bed sheets bed-side cupboard 2 0 2. Good hygienic conditions - Regular cleaning of the floors, access to portable water (less than 20 m) beds well spaced out (at least 1 2 0 m) - Ward well aerated without bad odors 3. Ward well lighted at night Electricity, solar energy, lamp with enough kerosene 1 0 4. Confidentiality assured Separate wards for men, women and children 1 0 5. Hospitalization register available and well filled-in Complete address and identity, reason for hospitalization, care received, duration of hospitalization, dates 1 0 of admission and discharge 6. Follow-up forms available and well filled-in and up-to-date - At least 10 un filled forms The evaluator verifies 5 filled (completed) forms :: 3 0  Temperature, BP, Pulse rate, Lab results filled in  Treatment follow-up ticked and conforms to protocol and treatment TOTAL POINTS - 10 points maximum …… / 10 XXXXXX Protocol 9. DRUGS AND SUPPLIES MANAGEMENT Protocol respected non respected 1. The staff indicate the security stock on the stock cards as = the average monthly consumption rate /2 Stock in the cards corresponds to the physical stock (real stock) 5 0  The evaluator takes a sample of three drugs 2. The health staff (health facility) has access to the accredited distribution (sales) centers for drugs, equipment and supplies known to the health district 1 0 3. The drugs are well arranged and well kept. Tidy room and well aerated with cupboards and shelves and drugs arrange according to class 2 0 and alphabetical order 83 4. The main pharmacy store (warehouse) supplies the facility sales point on a daily basis as per requested needs - The evaluator verifies if the quantities requested by the sales agent (clerk) correspond to the 2 0 quantities supplied (check the request slip co-signed by the two parties). 5. Absence of expired drugs or drugs with falsified tickets - The evaluator draws 3 drugs and 2 supplies by chance : - Expired drugs well separated from the rest of the drug 2 0 - A monthly inventory of expired drugs done and sent to the District Health Service (reception note available) or destroyed according to norms with a report written 6. Drugs sales point in the facility (pharmacy):  Drugs dispensed in sachets or small bags. 3 0  Tablets manipulated (dispensed using a spoon  Availability of portable water for taking first dose of drugs in the health TOTAL POINTS - 15 points maximum …… / 15 XXXXXX Available Available NO 10. TRACER DRUGS Security Stock = Monthly average consumption (MAC)/ 2 Yes < MAC / 2 > MAC / 2 1. Amoxicillin caps /tabs 500 mg 1 0 2. Amoxicillin syrup 250 mg/ 5ml 1 0 3. Artesunate tabs 50 mg – amodiaquine 200 mg 1 0 4. Cotrimoxazol tabs 480 mg 1 0 5. Diazepam 10 mg / 2ml – injectable 1 0 6. Iron – folic acid 200 mg + 25 mg 1 0 7. Mebendazol tabs 100 mg 1 0 8. Methergine/syntocinone amp 10 Units 1 0 9. Metronidazol tabs 250 mg 1 0 10. Paracetamol tabs 500 mg 1 0 11. Quinine tabs 300 mg and quinine injectable 1 0 12. ORS / oral sachet 1 0 13. Sterile gloves 1 0 14. Sterile gauze 1 0 84 15. glucose Solution 5% 1 0 TOTAL POINTS - 15 points maximum …… / 15 XXXXXX Protocol respected Protocol 11. MATERNITY not respected 1. Enough water and soap in the delivery room 1 0 Flowing water or at least 20 liters of water in a container 2. Delivery room lighted at night. Electricity, solar energy or lamp with enough kerosene 1 0 3. Correct disposal of waste in the delivery room - dustbin + security box for sharps + antiseptic solution 1 0 4. Delivery room in good condition Wall constructed with bricks, plastered and painted. Floor cemented without cracks, ceiling in good condition and well sealed, window shutters made of wood or glass and window blinds 1 0 mounted, Doors in good functional state with good locks 5. Availability of Partogram forms (at least 10) 1 0 6. Analyze the partograms of the last 5 deliveries - verify if partograms are correctly and completely filled-in 5 0 7. Blood pressure control during delivery BP taken at least once every hour during labor and recorded in the partogram : the evaluator 1 0 verifies 3partograms 8. Systematic APGAR score assessment during delivery filled in the partogram in the 1st, 5th and 10th minutes: the evaluator verifies 3 partograms 1 0 9. All the deliveries are conducted by a qualified staff (at least a brevete nurse) Identification of the staff that assisted the delivery from the names in the delivery registers 2 0 10. Availability of measuring scale for height, a fetoscope, and a suction pump in a mild antiseptic 1 0 solution or a manual or electric aspirator in good working condition. 11. Availability of Sterile gloves (at least 10 pairs) 1 0 12. Availability of at least 2 sterile delivery kits (boxes) Each with at least 1 pair of scissors and 2 forceps 2 0 85 13. Availability of an episiotomy box Catgut and non absorbable sutures, disinfectant, local anesthesia, sterile gauze. A sterile box 2 0 with needle holder needles toothed and non-toothed forceps 14. Delivery bed in good condition Table with a dismountable mattress and foot stands 2 0 15. A functional baby scale available The evaluator tries the functionality of the baby scale with an object. 1 0 16. New borne care material available Sterile suture or clamp of umbilical cord ligation, sterile umbilical bandage, 2 0 A mild antiseptic solution (applied on all new borne babies) 17. Bucket for used linen available 1 0 18. LLITN available and mounted on all the beds of the maternity 2 0 19. Beds with mattresses and beddings in good condition 1 0 20. Labor room appropriate FS : At least 2 beds with mattresses made of Macintouch and not torn 1 0 TOTAL POINTS - 30 points maximum …. / 30 XXXXXX Protocol 12. MINOR SURGERY Protocol respected not respected 1. Minor surgery room in good condition - Wall constructed with bricks, plastered and painted. Floor cemented without cracks, ceiling in 1 0 good condition and well sealed, 2. Examination bed available - Easily pliable with a flat mattress covered with Macintouch 0.5 0 3. Basic equipment available in the room: - Local Anesthesia available (at least 20 ml) - Drum with sterile gauze - Surgical kit with needle holder, toothed and non toothed forceps and a pair of scissors (at least 3kits) 2 0 - Sterile gloves (at least 3pairs) - Absorbable sutures (at least 2) - Surgical blades (at least 3) - Sterile linen in a drum 86 - Kidney dishes (at least 2) 4. Minor surgery register well filled-in and up to date. 0.5 0 5. Hygienic conditions ensured in the minor surgery room - Dustbin for septic wastes with cover 1 0 - Security box (for sharps) and used - antiseptic solution TOTAL POINTS - 5 points maximum …… / 5 XXXXXX Protocol 13.TUBERCULOSIS – Diagnostic Center Protocol respected not respected 1. Conditions put together for DOTS - Treatment forms, Registers and technical manual available 2 0 2. Slides carrier available 1 0 3. Diamond pencil available 1 0 Available Available NO Security Stock = YES < AMC / 2 Average Monthly Consumption (AMC)) / 2 > AMC / 2 4. Rifampicin-isoniazide-pyrazinamide: tab120+50+300mg - RHZE 1 0 5. Streptomycin 1 g (for cases of resistance) 1 0 6. Etambutol/ RHE comp 400 mg /RHE 1 0 7. Sputum cups 1 0 8. Slides and reagents available 2 0 TOTAL POINTS - 10 points maximum … / 10 XXXXXX Protocol Protocol 14. VACCINATION respected not respected 87 1. Cold Chain – Regular control of the cold chain (temp chart) and thermometer found in the refrigerator compartment - Refrigerator available – Temperature charts (book) available and filled-in twice a day – including the day of the assessment 3 0 - Temperature between 2 and 8degrees Celsius. - The evaluator verifies the functionality of the thermometer and ensures that the temperature is between 2 and 8 degrees Celsius on the thermometer. 2. Vaccines not out of stock (DPT+HepB+Hip, BCG, Measles + Yellow fever, OPV, TT) - Stock cards available and up-to-date 1 0 - The evaluator verifies the physical stock in the fridge which has to be correspond to the theoretical stock 3. The vaccines are well arranged in the Fridge - Freezer: - Non freezing compartment: 1st level :OPV – Measles- Yellow fever-BCG 2nd Level: - DPT+HepB, +Hib Pneumo, TT, 1 0 3rd level: - Solvent Lowest Level: un frozen accumulators - Absence of expired vaccines or converted control pastilles - Tickets on the vaccine vials visible 4. The state of the cold chain kerosene/gas : stock of at least 14 L :bottle of gas Solar/electric fridge: battery in good condition 1 0 5. Accumulators well frozen (at least 6) 1 0 6. Syringes available - auto disable – at least 30 1 0 - Solvent – at least 3 7. Waste collected in appropriate dustbins Security boxes available 1 0 8. Stock of vaccination cards or ‘our way to health’ cards with growth curves (EPI and IWF) At least 10 1 0 9. EPI Register well filled-in – or system of cards available System capable of identifying dropout cases and children completely vaccinated 1 0 10. Good waiting conditions for Infant 1 0 Enough benches and/or chairs and shelter from sun and rain 88 11. Cue using numbers by order of arrival 1 0 12. Available Salter Baby Scale in good working condition Scale gauged at zero + Carrier in good condition – not torn 1 0 TOTAL POINTS – 14 points maximum …. / 14 XXXXXXX Protocol Protocol 15. ANTE-NATAL CONSULTATION respected not respected 1. Weighing scale available and well regulated at zero 1 0 (weighing scale for ANC alone) 2. Measuring tape available and in good condition (figures lisible) 1 0 3. Determination of the nutritional status of all pregnant women at ANC 3 0 4. ANC Form (for the health facility) available and well filled-in Verify the last 5 forms Physical Exam : weight – BP – height – number of births – last menstrual period 4 0 Lab. Test : Albuminuria – Glucose –Hb Obstetrical exam done: Fetal Heart beat, Fundal height, presentation quickening 5. ANC Form (for the health facility) should show administration or prescription of iron folic acid and mebendazol 2 0 6. ANC Books (for mothers) available – at least 10 1 0 7. ANC Register available and well filled-in Mothers well identified, vaccination status, date of consultation, high risk pregnancy rubric filled-in 2 0 indicating problem and actions taken 8. Malnutrition detection book filled-in and up-to-date (number of ANC = number of investigations) 2 0 TOTAL POINTS – 16 points maximum …… / 16 XXXXXX Protocol Protocol 16. FIGHT AGAINST VIH/AIDS (VCT) respected not respected 1. Counseling room in good condition Walls constructed with bricks, plastered and painted. Floor cemented without cracks, ceiling in good condition and well sealed, window shutters made of wood or glass and window blinds mounted, Doors 2 0 in good functional state with good locks - Office with 3 chairs and shelves 89 2. Counseling register and results available and up-to-date 2 0 3. Skilled (trained) staff in place for VCT 2 4. Material for collecting samples available in the counseling room - tubes and needles (at least 20) 4 0 - tubes holder (rack) - tunicate 5. Optimal Hygienic conditions in the counseling room - Dustbin for septic waste with cover 2 0 - Secutiy box (for sharps) and well used 6. Condom available (at least 10) in the counseling 2 7. Security Stock for the reagents (at least 50 tests of détermine and at least 5 confirmation tests) with concordance between the physical and theoretical stocks 5 8. Respect of HIV testing (diagnosis) protocol (look at 10 cases in the VCT register) 6 0 TOTAL POINTS – 25 points maximum …… / 25 XXXXXX GRAND TOTAL SCORE IN PERCENTAGE 90 91 Appendix E OLS health care shopping results ANC in assigned treatment Skilled delivery in assigned Postnatal care in assigned group treatment group treatment group b se b se b se Post indicator 0.022 0.041 0.042 0.050 0.092 0.070 PBF/Post interact 0.060 0.053 0.047 0.062 0.031 0.086 Control 1/Post interact -0.001 0.053 -0.034 0.064 -0.083 0.090 Control 2/Post interact 0.015 0.052 0.037 0.063 -0.015 0.085 p-value PBF vs. C1 0.202 0.150 0.121 p-value PBF vs. C2 0.334 0.858 0.514 p-value PBF vs. C3 0.255 0.452 0.716 Post indicator 5161 4259 2087 * = p < 0.10, ** p < 0.05, *** p< 0.01 † Results from difference-in-differences regression models examining the effect of PBF on reproductive health service use among female respondents included in the household survey who had been pregnant in the previous 24 months. Regression model adjusted for individual (age, marital status, education level, religion, ethnicity, working status and type of work) and household control variables (number of individuals in the household, housing type, house ownership, water source, and type of sanitation) 92 Appendix F Results of household analysis with stratification on baseline facility bypassing Test for difference in Skilled delivery Above median bypassing Below median bypassing coefficients β se β se p-value Post indicator 0.080*** 0.029 0.022 0.02 PBF/Post interact -0.060 0.042 -0.027 0.034 0.5152 Control 1/Post interact 0.037 0.052 -0.004 0.034 0.4952 Control 2/Post interact -0.091** 0.039 -0.018 0.039 0.1721 p-value PBF vs. C1 0.069 0.565 p-value PBF vs. C2 0.443 0.841 p-value PBF vs. C3 0.149 0.431 N 2797 3128 ANC Above median bypassing Below median bypassing β se β se Post indicator 0.033 0.021 0.017 0.018 PBF/Post interact 0.030 0.030 -0.024 0.024 0.1488 Control 1/Post interact -0.040 0.025 -0.004 0.029 0.3343 Control 2/Post interact -0.051* 0.026 -0.04 0.028 0.7615 p-value PBF vs. C1 0.016 0.489 p-value PBF vs. C2 0.007 0.552 p-value PBF vs. C3 0.315 0.326 N 3301 2872 Postnatal care Above median bypassing Below median bypassing β se β se Post indicator 0.149*** 0.043 0.036 0.047 PBF/Post interact -0.034 0.052 -0.008 0.077 0.7785 Control 1/Post interact -0.108** 0.054 0.045 0.063 0.0579 Control 2/Post interact -0.085 0.053 -0.056 0.057 0.7022 p-value PBF vs. C1 0.102 0.484 p-value PBF vs. C2 0.254 0.511 p-value PBF vs. C3 0.516 0.914 N 3455 2447 93 Appendix G WHO well-being index (8) WHO well-being index Now I will read five statements about how a person might be feeling. For each of the five statements, please indicate whether in the last two weeks, you have been feeling this way most of the time, more than half of the time, less than half of the time, only rarely, or never. PLEASE SHOW AND ASK TO PICK OUT THE COLORED AND NUMBERED CARDS RESPONSE CODE MOST OF THE TIME 1 MORE THAN HALF OF THE TIME 2 LESS THAN HALF OF THE TIME 3 RECORD RESPONSE ONLY RARELY 4 CODE NEVER 5 (8.01) In the past two weeks, I have felt cheerful and in good spirits….. (8.02) In the past 2 weeks, I have felt calm and relaxed… (8.03) In the past 2 weeks, I have felt active and vigorous… (8.04) In the past 2 weeks, I woke up feeling fresh and rested… (8.05) In the past two weeks, my daily life has been filled with things that interest me…. 94 References Alkema, L., E. Broaddus, D. Chou, D. Hogan, C. Mathers, A.-B. Moller, L. Say, and S. Zhang. 2015. “Trends in Maternal Mortality: 1990 to 2015 Estimates by WHO, UNICEF, UNFPA, World Bank Group and the United Nations Population Division �. W. H. Organization: World Health Organization Basinga, P., P. J. Gertler, A. Binagwaho, A. L. B. Soucat, J. R. Sturdy, and C. M. J. Vermeersch. 2010. “Paying Primary Health Care Centers for Performance in Rwanda.� Policy Research Working Papers. Binyaruka, P., E. Patouillard, T. Powell-Jackson, G. Greco, O. Maestad, and J. Borghi. 2015. “Effect of Paying for Performance on Utilisation, Quality, and User Costs of Health Services in Tanzania: A Controlled Before and After Study.� PLoS One 10(8): e0135013. Bonfrer, I., R. Soeters, E. Van de Poel, O. Basenya, G. Longin, F. van de Looij, and E. van Doorslaer. 2014a. “Introduction of performance-based financing in Burundi was associated with improvements in care and quality.� Health Affairs 33(12): 2179-87. Bonfrer, I., E. Van de Poel, and E. Van Doorslaer. 2014b. “The effects of performance incentives on the utilization and quality of maternal and child care in Burundi.� Social Science & Medicine 123: 96-104. Bove, A., P. Basile., P. J. Robyn, and R. Singh. 2013. “Towards Greater Equity: A Special Focus on Health. Cameroon Economic Update �. T. W. Bank. Washington DC: The World Bank. Central Intelligence Agency. 2016. "Cameroon. In The world factbook." [accessed on December 8, 2016]. Available at: http://libanswers.brenau.edu/a.php?qid=710633. Falisse, J.-B., J. Ndayishimiye, V. Kamenyero, and M. Bossuyt. 2014. “Performance-based financing in the context of selective free health-care: an evaluation of its effects on the use of primary health-care services in Burundi using routine data.� Health Policy and Planning: czu132. Fritsche, G. B., R. Soeters, and B. Meessen. 2014. Performance-based financing toolkit: World Bank Publications. Gertler, P. J. and P. I. Giovagnoli. 2014. “Rewarding provider performance to enable a healthy start to life: evidence from Argentina's Plan Nacer.� World Bank Policy Research Working Paper(6884). Huillery, E. and J. Seban. 2014. “Performance-Based Financing, Motivation and Final Output in the Health Sector: Experimental Evidence from the Democratic Republic of Congo.� ICF International. 2012. "The DHS Program STATcompiler" [accessed on June 10, 2015, 2012]. Available at: http://www.statcompiler.com. Institut National de la Statistique. 2015a. “Enquête par grappes à indicateurs multiples (MICS5), 2014, Rapport de résultats clés.� Yaoundé, Cameroun: Institut National de la Statistique. Institut National de la Statistique. 2015b. “Enquête par grappes à indicateurs multiples (MICS5), 2014, Rapport Final.� Yaoundé, Cameroun: Institut National de la Statistique. Institut National de la Statistique. 2015c. “Rapport tendances-profil et détermiants pauvreté entre 2001 et 2014.� I. N. d. l. Statistique. Yaounde, Cameroon. Institut National de la Statistique (INS) et ICF International. 2012. “Enquête Démographique et de Santé et à Indicateurs Multiples du Cameroun 2011.� I. International. Calverton, Maryland, USA: ICF International Ir, P., C. Korachais, K. Chheng, D. Horemans, W. Van Damme, and B. Meessen. 2015. “Boosting facility deliveries with results-based financing: a mixed-methods evaluation of the government midwifery incentive scheme in Cambodia.� BMC Pregnancy & Childbirth 15(1): 1. Ministere de la Sante Publique. 2016. “Analyse Situationnelle du financement de la santé au Cameroun: Stratégie de Financement de la Santé 2017-2021.� M. d. l. S. Publique. Yaounde, Cameroon. 95 The United Nations Children's Fund. 2015. “Enquête par grappes à indicateurs multiples (MICS5) 2014 Rapport de résultats clés.� Multiple Indicator Cluster Survey (MICS). T. U. N. C. s. Fund. The World Bank. 2013a. “Cameroon Performance-based Financing: Results from the health facility baseline survey: Results from the health facility baseline survey.� T. W. Bank. Washington DC: The World Bank. The World Bank. 2013b. "Public randomization ceremonies" [accessed on December 18, 2016, 2013b]. Available at: http://blogs.worldbank.org/impactevaluations/public-randomization-ceremonies. The World Bank. 2014. “Impact Evaluation on PerformanceBased Financing In Haut-Katanga District, Democratic Republic Of The Congo.� World Bank Policy Note. T. W. Bank. Washington DC. The World Bank. 2016a. "Cameroon Country Overview" [accessed on December 8, 2016a]. Available at: http://www.worldbank.org/en/country/cameroon/overview. The World Bank. 2016b. "Health expenditure per capita (current US$)" [accessed on December 8, 2016b]. Available at: http://data.worldbank.org/indicator/SH.XPD.PCAP. The World Bank. 2016c. “World Development Indicators.� Washington DC, USA: The World Bank. United Nations Statistical Division. 2015. "Millennium Development Indicators: Country and Regional Progress Snapshots" [accessed on 2015]. Available at: http://mdgs.un.org/unsd/mdg/Resources/Static/Products/Progress2015/Snapshots/CMR.pdf. Zeng, W., M. Cros, K. D. Wright, and D. S. Shepard. 2013. “Impact of performance-based financing on primary health care services in Haiti.� Health Policy Plan 28(6): 596-605. 96