2021 Community-based Care Quality Study Report INTRODUCTION........................................................................................................................................................... 2 The community health worker program in Mozambique ................................................................................. 2 Review of selected community health worker programs................................................................................. 3 Study objectives ...................................................................................................................................................... 6 METHODS...................................................................................................................................................................... 6 Survey design .......................................................................................................................................................... 6 Sampling ................................................................................................................................................................... 7 Data collection and analysis ................................................................................................................................. 8 RESULTS ......................................................................................................................................................................... 8 General characteristics .......................................................................................................................................... 8 Quality of care ..................................................................................................................................................... 10 Clinical vignettes .............................................................................................................................................. 10 Health prevention and promotion ................................................................................................................ 13 Medicines and supplements ........................................................................................................................... 14 Community engagement................................................................................................................................. 15 User satisfaction ................................................................................................................................................... 16 Factors influencing APE performance............................................................................................................... 17 Variable selection and model ....................................................................................................................... 17 Results ................................................................................................................................................................ 18 CONCLUSIONS......................................................................................................................................................... 19 Appendix A: Descriptive statistics of APEs and supervisors ............................................................................. 21 Appendix B: Detailed responses to medical vignettes ...................................................................................... 22 Appendix C: APE medicine kit list and categorization used ............................................................................ 25 Appendix D: upSCALE in Inhambane and Zambezia ........................................................................................ 26 1 INTRODUCTION The community health worker program in Mozambique Community involvement for promoting and providing health services is one of the principles that underpin the Mozambique Health Sector Strategic Plan (2014–2024) and the Investment Case.1 2 Strengthening community-based health services is one of the key interventions to ensure access and utilization of available health services. The National Strategy guides its implementation for Health Promotion (2015– 2019). Community-based health services in Mozambique are primarily provided through the Agentes Polivalentes Elementares (APEs), embedded in the decentralized health structures. Existing since 1974, the Ministry of Health (MOH) revitalized its APE program in 2010 after setbacks encountered during the civil war (1976–1992).3 The revitalized program, led by the Department of Health Promotion under the National Directorate of Public Health, sought to increase the coverage and the quality of services provided, aiming to train and deploy additional APEs across the country, with a primary role in health promotion and disease prevention (80 percent of their time) and a secondary role in curative services (20 percent of their time).4 APEs were designated for training in a revised national five-month training package provided by the provincial supervisor and the district supervisor. Subsequently, APEs are attached to a health facility that is 8–25 km from their designated communities, serving a population of 500–2,000 individuals, with an average of 25 per district.5 Refresher trainings are planned for every two years, with ad hoc trainings on major health threats to the countries organized. APEs primarily utilize a door-to-door approach and interact with other community- and facility-based structures, including the health committees (comités de saúde) and co-management committees (comités de co-gestão). A facility-based supervisor oversees their work. Although APEs are officially considered volunteers, they receive a monthly stipend of 1,250 meticais (approximately US$20) and are provided with a bicycle, working kit, and monthly kit of medicines. The medicine kit (see Appendix C) includes basic medicines for first aid and common ailments such as diarrhea and acute respiratory infections. Since 2013, the kit is complemented by rapid diagnostic tests for malaria and four kinds of formulations of Atriméter-Lumefantrina as an antimalarial treatment. Contraceptives are provided separately through the pharmacy at the health facility. Although there are no minimum requirements for the APE’s monthly work, the kit contains supplies for about 250 possible consultations.6 An APE’s work and use of medicines are recorded through a monthly consultation report and stock management sheet. Across Mozambique, APEs are estimated to serve about 12 million people, roughly 40 percent of the country’s population. In 2020, they carried out approximately 5 million consultations; conducted 2.6 million rapid diagnostic tests (RDTs), confirming approximately 1.8 million cases of malaria; and identified approximately 334,494 cases of diarrhea and 409,893 cases of pneumonia in children under five. APEs provided 1.6 million doses of the antimalaria medication “Coartem� and 30,991 of Artesunate in 2020. They also engaged in health promotion and disease prevention, conducting over 10 million house visits, including approximately 600,000 for women during pregnancy or postnatally, 478,000 visits for newborns, and 3.9 million visits for children under five. APEs provided a second dose of Vitamin A to 677,125 children and chlorhexidine to 48,994 newborns.7 As part of the Primary Healthcare Strengthening Program (PHCSP),8 which supports the Investment Case, Disbursement-Linked Indicator 10 (DLI10) incentivizes both the increase in the APE workforce and 1 Plano Estratégico do Sector da Saúde, 2013. 2 Investment case of the Republic of Mozambique, 2017. https://www.globalfinancingfacility.org/investment-case-republic- mozambique. 3 Chilundo B. G., Cliff J. L., Mariano A. R., Rodríguez D. C., and George A. Relaunch of the official community health worker programme in Mozambique: is there a sustainable basis for iCCM policy?. Health Policy Plan. 2015; 30 Suppl 2 (Suppl 2):ii54–ii64. doi:10.1093/heapol/czv036. 4 MISAU, Pontos Chave para a Implementação do Programa dos Agentes Polivalentes Elementares. 2010. 5 MISAU, Pontos Chave para a Implementação do Programa dos Agentes Polivalentes Elementares. 2010. 6 UEM, UNICEF, Avaliação do sistema de abastecimento de medicamentos para os Agentes Polivalentes Elementares em quatro distritos de Moçambique no período de Fevereiro de 2019 a Agosto de 2020. October 2020. 7 Data for APEs’ activities provided by UNICEF; population data from the World Bank WDI, 2019. 8The Primary Healthcare Strengthening Program is managed by the World Bank and cofinanced by the Global Financing Facility, Canada, the Netherlands, the UK Foreign and Common and Development Office (FCDO), and USAID. 2 improvements in the quality of their services. APEs are considered key agents to improve the utilization and quality of reproductive, maternal, child, and adolescent health and nutrition services in underserved areas. The support provided under the PHCSP is expected to increase the number of active APEs from 3,380 in 2017 to 8,800 by 2023, and 80 percent of APEs are expected to meet a minimum quality standard established. Review of selected community health worker programs The recognition of community-based health workers (CBHW)9 as vital to the improvement of primary health care by the Declaration of the Alma Ata in 1987 drove an increase in the interest in CBHW programs.10 Initial programs faced the following difficulties: (1) inadequate training, remuneration, and incentives; (2) limited supervision; (3) deficient continuing education opportunities, inadequate supplies, and medicines; and (4) limited recognition of acceptance by other health workers.11 These difficulties led to a decline in the interest in CBHW programs in the late 1980s. However, CBHW programs reemerged in the mid-1990s in low- and middle-income countries as a tool to address the demands of growing HIV and infectious diseases; inequalities in the access to health services, particularly in rural areas; and thanks to increased global advocacy, to reduce the burden on overstretched and understaffed health systems.12 Policies and approaches behind the different community health strategies vary considerably across low- income countries. Comparing these can be useful to inform changes in national CBHW strategies. As part of this study, the experiences from Ghana, Ethiopia, and Rwanda are considered and compared to those of Mozambique. The four Sub-Saharan countries have a comparably low Human Capital Index (0.36– 0.45), high percentages of the rural population (43 percent–83 percent), and substandard maternal and child health indicators, including high fertility levels, maternal mortality, under-five mortality, and stunting. In addition, Mozambique is disproportionately more affected by the HIV/AIDS epidemic (see table 1 and table 2). Table 1: Key Health Indicators: Mozambique, Ghana, Ethiopia, and Rwanda, World Bank Indicators Mozambique Ghana Ethiopia Rwanda GDP per capita (current US$) (2019) 503 2,202 820 855 Total population (million) (2019) 30.36 30.41 112.07 12.62 Rural population (%) (2019) 63% 43% 79% 83% Human Capital Index (2020) 0.36 0.45 0.38 0.38 Total fertility rate (TFR) (2019) 4.80 3.8 4.1 4.0 mCPR % (15–49 years)13 25% 24% 41% 48% MMR (per 100,00 live births)14 589 334 557 297 Under mortality rate (per 1,000 live births) (2019) 74 46 51 34 Stunting rate for children under 5 (%)15 42.3% 17.5% 36.8% 33.1% HIV prevalence rate (% population ages 15 to 49) (2019) 12.4% 1.7% 0.9% 2.6% Note: MMR = maternal mortality rate. mCPR = modern contraception prevalence rate 9 CBHW is used as a broad term in scope including home-based care providers, community health works, community-based treatment supporters, and traditional birth attendants. 10 Liu A., Sullivan S., Khan M., Sachs S., and Singh P. 2011. Community health workers in global health: scale and scalability. Mt Sinai J Med, 78(3):419–435. 11 Zulu J. M., Kinsman J., Michelo C., and Hurtig A. K. 2014. Integrating national community-based health worker programmes into health systems: a systematic review identifying lessons learned from low- and middle-income countries. BMC Public Health. September 22, 2014;14:987. doi:10.1186/1471-2458-14-987. PMID: 25245825; PMCID: PMC4192351. 12 Glenton C., Colvin C. J., Carlsen B., Swartz A., Lewin S., Noyes J., and Rashidian A. 2013. Barriers and facilitators to the implementation of lay health worker programmes to improve access to maternal and child health: qualitative evidence synthesis. Cochrane Database Syst Rev, (10): Art. No.: CD010414. doi:10.1002/14651858.CD010414.pub2. 13 Data for 2015 for Mozambique and Rwanda. Ghana 2018. Ethiopia 2019. 14 Data from 2011 for Mozambique; 2015 for Rwanda; 2017 for Ghana; and 2016 for Ethiopia. 15 Data from 2015 for Mozambique; 2017 for Ghana; 2019 for Ethiopia; and 2020 for Rwanda. 3 Table 2: Key Metrics of CBWH Programs: Ghana, Ethiopia, and Rwanda Mozambique Ghana Ethiopia Rwanda Type of provider Elementary Community Health Female Health MCH agents multipurpose Officers (CHOs) Extension Workers female/male agents (APEs) (salaried). (salaried). tandem (salaried (salaried, volunteers). performance- based). Coverage ratios 1:500–2,000 Up to 3 CHO per 1: 2,500 people 3:300 – 450 individuals 5,000 people. people Selection criteria Age > 18. Trained community Grade 10. Primary school Basic skills and health nurse. completed. arithmetic. Age > 18. Ability to read and Ages 18 to 30. Ages 20 to 50. write in Portuguese. Training + roles 5-month training. 2 years training for 1 year, including 15 days for MCH Refreshers planned community nurse 320 hours of agents for every 2 years. certificate + 2 practical Routine activities weeks on-the-job experience. 27 days for include developing training. “binôme.� strong ties to their Refresher every 2 community, health Integrated minimum years. On-the-job training promotion and service package of and periodic education, family promotion and Integrated minimum refreshers. planning counseling, prevention. service package and prevention and including FP, MCH, Differentiated treatment of Districts can nutrition, NB care, roles: common ailments. customize training HIV and TB, MCH + deliveries; manuals and include Malaria and Treat common additional WASH. Case management, ailments, including services/roles with pneumonia, childhood illnesses: diarrhea, support from diarrhea; pneumonia, NGOs. Common health malaria, minor promotion themes injuries, worms, and others. (TB DOTs, GBV screening, FP, immunization, hygiene). Community Collaborate with Community health “Model households� Community Health support health and volunteers provided in defined networks Cooperative community co- with basic training, of HH, with trained support outreach management bicycles, and social female member workers, oversee committees comités recognition. support extension performance, de saúde and workers (health entrusted with comités de development army). payments. cogestão. 70% payment used for community projects. Note: DOTs = directly observed treatments for (TB); FP = family planning ; GBV = gender-based violence; HH = household; MCH = maternal and child health ; NB = newborn; NGOs = nongovernmental organizations; TB = tuberculosis; WASH = water, sanitation, and hygiene. The case of Ghana16 16 Community health systems catalogue, country profile: Ghana. USAID. 2017. 4 Ghana’s model was based on an initial pilot where four service delivery models were tested, and the most successful model defined the structure of the national CBHW program. It consists of two cadres of community health workers, including salaried Community Health Officers (CHOs) and Community Health Volunteers (CHVs). CHOs live and work in a geographical area with 750 households or 5,000 people. They work closely with community leaders and other cadres of community health workers operated by nongovernmental organizations (NGOs). CHOs are trained for two years as community health nurses and provided with on-the-job training for two weeks on CHO functions. They are required to have at least one year’s experience working in a health center and be 18 to 30 years old. Other skills such as planning, communication, and interpersonal skills are used as part of the selection criteria. CHOs deliver a predefined package of essential services, including health promotion and disease prevention, family planning, reproductive health, maternal and child health, immunization, and the treatment of diarrhea, malaria, acute respiratory infections, and childhood illness, among others. CHVs assist with household visits and are supervised by the CHOs. Paper-based forms for collecting data and a community register are kept, along with tally cards which are provided by the subdistrict health authorities, that are used to track the use of medicines and supplies. The case of Ethiopia17 Ethiopia’s Health Extension Program uses a grassroots approach in which households with “model women� lead their families and surrounding households to implement and monitor health plans. Model women are supported by female community health extension workers (HEWs) who work closely with health posts and cover a network of up to 30 households led by model women. HEWs are recruited from their communities, and their one-year training includes practical experience and is recognized within the National Vocational Education and Training System. HEWs provide a package of services that include communicable and non-communicable diseases, maternal and child health (MCH)care, including sexual and reproductive health and rights (SHRS), malaria case management, immunization, and emergency response. Records of their work are kept in a family folder, including a series of health cards that track health hand demographic information that feeds into the national health management system. Supplies are obtained using an official request procedure from the health facility. The Ethiopian model is government-led and financed. Health partners and NGOs support the implementation of this national program and provide technical and financial assistance upon request from the government and in alignment with the Health Sector Manual. The case of Rwanda18 The Rwanda model consists of two cadres of community health workers (CHWs): Agents de Santé Maternelle (ASMs, maternal health agents) and binômes (tandems). The village elects the two different cadres of health workers, who then receive training adapted to their different responsibilities. ASMs focus on maternal, neonatal, and child health services. In contrast, the two types of tandems (one male, one female) focus on integrated community case management for malaria, diarrhea, pneumonia, and monitoring of malnutrition and providing family planning. CHWs receive on-the-job training and periodic refreshers organized by the Ministry of Health. Their work is supported by community groups, including cooperatives, parents’ groups, and water, sanitation, and hygiene (WASH) clubs. The three types of CHWs are remunerated through a community performance-based financing system, where 30 percent of the incentives are provided to the CHWs, and 70 percent are provided to the CHW cooperative that invests in income-generating activities. Data from their activities are collected through an electronic information system that is consolidated nationally by a dashboard. CHWs use mobile phones to report on specific pregnancies in real-time and collect data using reports and stock cards consolidated at the village level. Determinants for positive integration of CHW programs into the national health system Compared to the three cases presented, the Mozambican CHWs are smaller in number, cover large populations, and are less supported by predefined/trained community-based representatives. In addition, the program depends heavily on external donor funding, with NGOs having separate agents for specific interventions or vertical programs. CHWs in other countries are either more qualified upon recruitment or receive more extended training. Performance-based allocation or a financial retribution 17 Community health systems catalogue, country profile: Ethiopia. USAID. 2017. 18 Community health systems catalogue, country profile: Rwanda. USAID. 2017. 5 system that is more inclusive of community needs and efforts does not occur in Mozambique. In addition to these different metrics, a systematic review of 36 studies related to CBHW developed a conceptual framework to categorize the determinants that support a positive integration of CBHW programs into national health systems and provided the following conclusions.19 Important findings and considerations include: • Careful planning and funding: The gap in human resources for health in low- and middle- income countries is often one of the key factors that contribute to the integration of CHW into the health systems, as CBHWs have the potential of extending services to hard-to-reach areas and can play an important role in achieving demonstrable health benefits. The pathway to scale and integration has often been limited by planning and logistics management and insufficient and inconsistent program funding. • Ability to deliver good services: Good services and improved health outcomes can enhance acceptability and adoption of CBHWs by the population and preference over other traditional and community-based actors. Quality of services is influenced by workload, training, supervision, and effective payment structures. • Perspectives to adopt the system: Positive perceptions from politicians, communities, and health workers are key to integration. Lack of clarity regarding roles or hierarchy can create resistance from nurses or health personnel. Community perceptions are also important, and recruitment from local people has been shown to be positive. Still, considerations regarding gender and cultural norms are also important to consider as part of the recruitment criteria. For instance, mothers’ utilization and acceptance of MCH and FP services have been associated with having women CHWs in Ethiopia and India, with whom women could share their personal stories. • Compatibility with existing systems: Building a CHW program into existing data systems, payment structures, and hierarchies/communication lines facilitates the expansion and integration into the national health system and program compatibility with local practices, values, and regulations. It can also foster relationships with other health personnel and ease referral processes and general service delivery. Study objectives This study is undertaken as part of the advisory services and analytics (ASA) that accompanies the implementation of the PHCSP. It builds on previous analytical work on APEsand discussions with the APE technical working group, which identified constraints in the program’s functioning, including stockouts, supervision, and recruitment. This ASA seeks to assess the QoC provided by APEs to inform policymaking for the APE program from an evidence-based perspective. Specifically, the objectives of the study are to: 1. Assess the quality of primary health care services provided at the community level in Mozambique; 2. Establish a baseline for quality of care against which quality of care can be benchmarked and improved throughout the implementation of the PHCSP; and 3. Identify and assess contributing factors to successes and failures in the APE program across all provinces in Mozambique. METHODS Survey design The study conducted a nationally representative, cross-sectional quality of care (QoC) survey of APEs and providers in primary health care facilities across Mozambique. It assesses not only the knowledge and competency of APEs but also the network of technical support (supervisors) and supplies available to them, using four survey instruments (see table 3): 19Zulu, J. M., Kinsman, J., and Michelo, C., et al. 2014. Integrating national community-based health worker programmes into health systems: a systematic review identifying lessons learned from low- and middle-income countries. BMC Public Health 14, 987. https://doi.org/10.1186/1471-2458-14-987. 6 (1) General questionnaire: a simple introductory questionnaire for community health workers or Agentes Polivalentes Elementares (APEs) and supervisors (age, position, years in service, relevant training received, among others); (2) Medical vignettes: hypothetical medical cases posed to the APEs and supervisors and a structured questionnaire used to record the specific history-taking questions and examinations reported and any articulated diagnoses and prescribed treatment, along with a complementary section regarding knowledge of, and engagement in, health promotion activities; (3) Direct observation checklists: enumerator checks for the contents of an APE’s drug kit and other supplies, and availability of functioning equipment including mobile phone application; and (4) Household and client surveys: short questionnaires administered to clients by telephone, selected randomly from the APE’s list of phone numbers for all rural members in their catchment area. Table 3: Overview of Survey Instruments • Data on age, years of service, relevant training, etc. (1) General questionnaire • Targeted APEs and APE supervisors. • Four case studies to evaluate whether APEs can: assess medical history, ask appropriate questions and conduct relevant exams, make a diagnosis and prescribe/administer adequate treatment. (2) Medical vignettes • Targeted APEs and APE supervisors. • This section also contains complementary questions aimed at assessing preventative services offered by APEs. • Data on the content of APEs’ medicine kit and work equipment. (3) Direct observation checklists • The verification took place before the APE obtained a refill at the reference health facility. (4) Household and client • Short questionnaire to users of health services in the catchment area of the survey sampled APEs. Sampling The population of interest for this study is: • APEs from all provinces in Mozambique;20 • Rural households, APE clients; and • APE supervisors from an APE’s reference health facilities. The study sample included any individuals who satisfy the following conditions: • APEs across the country, randomly selected from a list provided by the reference health facilities; • Rural households, APE clients, randomly selected from a phone list provided by the APEs in the sample, over 18 years of age; • APE supervisors working in the reference health facilities of the APEs in the sample; and • Other health personnel in the reference health facilities. The sampling frame was based on the health facilities to which the APEs are mapped. Health facilities were chosen using probability sampling, except for Cabo Delgado21 and Inhambane, to capture the effect of the upSCALE program, which began in 2019 in these provinces. The number of health facilities was inferred from the total number of APEs in each province, stratified as: 1. 1–3 APEs per health facility; 2. 4–6 APEs per health facility; and 20 Maputo City is not included, as APEs in this province are not attached to a reference health facility; Cabo Delgado was subsequently excluded from the study due to safety concerns in data collection. 21 Originally included in the sampling. 7 3. > 7 APEs per health facility. All APEs attached to the selected health facilities were included in the sample. The final number of APEs interviewed was 1,377, roughly 24 percent of the total number of APEs in Mozambique at the time of the survey design. All APE supervisors who were available were interviewed for each health facility, and the final sample included 320 supervisors. Finally, health users were randomly selected from the list provided by each APE. Three users over 18 years of age were selected for each APE. The final number of users surveyed was 2,404. Data collection and analysis Data was collected using the four survey instruments described in the “Survey design� section. After initial training in each region, data collection was carried out simultaneously in Mozambique’s three regions (North, Central, Southern). Each team was assigned a supervisor who oversaw the first week of data collection. A team leader was present throughout the data collection process, a field supervisor who performed quality control, five enumerators, and one driver. Data was collected between September and December of 2020. Data collection was carried out using Computer-Assisted Personal Interviewing (CAPI) with a Census and Survey Processing System (CSPro) for Android tablets. CSPro is an open-source software that enables data collection, cleaning, management, and processing. Once surveys were filled out, they were downloaded to a central server. After this process was complete, questionnaires were reviewed for data validation or rejection by a supervising team based in Maputo City. A rejection of a survey would trigger either a correction or retake. Data analysis includes producing descriptive statistics of demographic data for APEs, their supervisors, and rural households who use community health services, computing indicators for APEs’ knowledge and community engagement, availability of medicines and supplements, the frequency of training and supervision;, and identifying potential factors driving APE performance through multivariate regression analysis. RESULTS General characteristics In 2020, the Community Health Workers program in Mozambique was comprised of an estimated 6,690 Agentes Polivalentes Elementares, or APEs, and 1,398 health workers who supervised them. Based on the responses to this study, one APE serves on average between 600 and 700 households, a higher load than that of other Sub-Saharan African countries with similar programs, such as Ghana (3:5,000 individuals), Ethiopia (1:2,500 individuals), or Rwanda (3:300–450 households). Based on the nationally representative sample, a typical APE would be a man (60 percent of sampled APEs were male), in his mid-thirties, with an average of five years of experience as an APE and a primary school education. On average, APEs reported serving between 600 and 700 households and spending about 17 hours per week in this role. A typical supervisor would be a nurse or medical technician, male (69 percent) and in their early thirties, with two years of experience as an APE supervisor. Supervisors self-reported spending, on average, five hours per week on supervision (see Figure 1). Detailed descriptive statistics can be found in Appendix A. 8 Figure 1: Profile of the Average APE and Supervisor • Mean age 36 • Mean age 31 • Male • Male • Completed primary schooling • Nurse/medical technician • 5-year experience • 2-year experience • Dedicates 17 hours/week • Dedicates 5 hours/week • Serves 600–700 households • Supervises 5 APEs APE Supervisor Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Training is an important aspect of the community health workers’ program. In addition to their initial five- month training and ongoing supervision, APEs are expected to receive a refresher training every two years on relevant topics for community health. The last APE refresher was carried out in 2016, and a new training based on a revised curriculum planned for 2019 was subsequently postponed because of the COVID-19 restrictions. The survey revealed that 96 percent of APEs had received at least one such course, and, on average, APEs received refresher courses in five topics throughout their service. The most common subject for additional training was malaria, followed by family planning and diarrhea, as shown in Figure 2. Conversely, less than 50 percent of the APEs surveyed had received refresher courses on pneumonia and respiratory infection (49 percent), newborn care (46 percent), pregnant women (45 percent), and HIV management (31 percent). Figure 2: Percentage of APEs Who Reported Having Received Refresher Courses after Initial Training 90 80 70 Percentage of APEs 60 50 40 30 20 10 0 Malaria Family Diarrhea Nutrition Water and Pneumonia/ Newborn Pregnant HIV planning sanitation respiratory care women management infection Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. While APEs did not specify when they last received the refresher courses, they reported attending on average one training session over the previous three months (Figure 3). More than half of APEs in Nampula, Zambezia, Manica, and Inhambane had not received any training over the previous three months. The result is likely to include refresher trainings that were rolled out with the support of partners on malaria (2018/2019) and nutrition (2019/2020) 9 Figure 3: Percentage of APEs Who Attended a Training Session over the Previous Three Months 70 60 Percentage of APEs 50 40 30 20 10 0 Tete Sofala Gaza Niassa Maputo Inhambane Nampula Zambezia Manica Province Did not attend 1 session more than 1 session Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Another essential feature of the community health program is supervision by health personnel. The physical verification of APEs’ and supervisors’ reports was conducted. APEs’ records revealed that they had received on average one supervision visit over the previous three months, but 28 percent did not receive any. Twenty-nine percent of APEs received one supervision visit per month. In Inhambane, 65 percent of APEs hadn’t registered any supervision sessions over the previous three months. The verification of supervisors’ registers showed similar deficiencies. While most APE supervisors had updated records of the monthly activities performed by their APEs, less than 50 percent had evidence of their supervision of APEs’ work in the community, and only 57 percent had evaluated APEs’ clinical competencies (see Table 3). Table 3: Verification of APE Supervisors’ Checklists APE supervisors’ checklists Number Percent Summary sheet of monthly activities of APEs is available 270 84 Summary sheet of monthly APE activities is completed for the past three months 243 76 Verification list for APE supervision in the community is available 188 59 Verification list for APE supervision in the community is completed for each of the APEs 152 48 over the past three months Evaluation of APE’s clinical competencies is completed for each APE over the past three 182 57 months Quality of care Clinical vignettes The use of clinical vignettes to assess the quality of care is well documented in developed and developing countries alike.22 Clinical vignettes facilitate standardized comparisons and offer a pragmatic method for assessing provider care quality. They have been validated against the “gold standard� of unannounced standardized patients, with the advantage of being more cost-effective. In fact, they have been shown to be more accurate than medical records examinations.23 Limitations to their use in developing countries include potential misalignments with inputs provided in actual consultations.24 Four 22 See, for example: Peabody, John W., and Anli Liu. 2007. "A cross-national comparison of the quality of clinical care using vignettes." Health Policy and Planning 22.5: 294 –302. 23 Peabody, John W., et al. 2004. "Measuring the quality of physician practice by using clinical vignettes: a prospective validation study." Annals of internal medicine 141.10: 771 –780. 24 Leonard, Kenneth L., and Melkiory C. Masatu. 2005. "The use of direct clinician observation and vignettes for health services quality evaluation in developing countries." Social science & medicine 61.9: 1944–1951. 10 cases of common ailments were presented to the APEs as clinical vignettes, and basic symptoms and patient characteristics, such as age and gender, were described. In two of the cases, APEs were instructed to ask questions that would help them identify the ailment. Additional information was provided to the APEs (irrespective of the questions asked), and APEs were then requested to describe the diagnosis for the symptoms provided. Subsequently, APEs were asked what the appropriate treatment would be. The cases described a scenario of diarrhea and pneumonia in children, a case of malaria in children, one describing potential signs of danger during pregnancy, and one of malaria in adults (see Figure 4). The third case was subsequently excluded from the study as it was deemed too complex and was not clearly understood by a sufficient number of interviewees. Figure 4: Cases Described to APEs in Clinical Vignettes Case 1 • 4-month-old with diarrhea for 2 days with no blood; cough for 5 days and breathing of 56 respirations/minute Case 2 • 10-month-old with a temperature; has not eaten in 3 days and cannot stay awake Case 3 • 35-year-old pregnant woman with headache and stomach pain Case 4 • 26-year-old with a positive RDT malaria result Source: Quality of Care (QoC) survey of APE, 2020. For the purposes of this analysis, an APE was classified as having diagnosed the common ailment correctly if, in addition to providing an accurate diagnosis, they did not also identify an incorrect diagnosis (multiple answers were allowed). A treatment was considered accurate when APEs mentioned any correct treatment actions (and no incorrect actions) after correctly diagnosing the case. Figure 5 depicts the share of APEs who correctly identified and recommended the appropriate treatment for each case. Among the clinical cases, APEs performed worse in identifying signs of danger in pregnancy (19 percent of APEs correctly diagnosed this case), followed by respiratory infection and diarrhea in children (48 percent). The probability of a correct diagnosis for pneumonia and diarrhea in children was highest in Gaza (87 percent) and lowest in Zambezia (24 percent). Less than 5 in 20 children in Zambezia would be correctly identified as having a condition consistent with experiencing cough, rapid breathing, and diarrhea. The share of APEs who correctly identified malaria in children was above 50 percent in all provinces, except in Zambezia, where it was only 36 percent (see Figure 5). An adult with malaria and a positive rapid diagnostic test (RDT) result had approximately a 65 percent chance of being accurately diagnosed, but that chance rose to 100 percent in Maputo Province. Figure 5: Percentage of APEs Who Correctly Identified and Treated Case Scenarios 11 The results are considerably worse when it comes to APEs’ ability to prescribe an appropriate treatment or course of action to the cases described. This result is in part due to the fact that APEs who were not able to correctly identify the clinical case were excluded from the second step of this analysis. Nevertheless, even among those APEs who correctly identified the clinical case, less than 50 percent identified the appropriate actions to take.25 In order to establish a baseline measure for APE knowledge, the number of correct diagnoses and treatments were counted for each APE. Case 3 in Figure 4 (pregnant woman with headache and stomach pain) was dropped from this metric due to concerns over lack of clarity in the case description and the lack of a clear diagnosis, which would be out of the scope of APEs’ training and not a fair assessment of their knowledge. Overall, 28 percent of APEs were able to identify all three cases correctly, and only 4 percent recommended appropriate treatment for all three cases (see Figure 6). On average, APEs correctly identified 1.7 cases and prescribed an appropriate course of action for one case. Figure 6: Percentage of APEs Who Correctly Diagnosed Case (left) and Recommended Appropriate Treatment (right) 48% 32% 28% 24% 28% 16% 20% 4% 0 cases 1 case 2 cases 3 cases 0 cases 1 case 2 cases 3 cases Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Significant differences were observed among provinces regarding APEs’ ability to identify clinical cases correctly (Figure 7). In Maputo Province and Gaza, all APEs were able to correctly identify at least one of the three clinical cases presented, with Niassa (99 percent) and Sofala (98 percent) showing similar performances. Conversely, 31 percent of APEs in Zambezia and 24 percent in Tete could not correctly identify a single case. Figure 7: Average Number of Cases APEs Correctly Diagnosed and Treated by Province Diagnosis Treatment 2.4 1.6 2.3 2.2 2.2 1.5 2.0 2.0 1.4 1.9 1.7 1.1 1.1 1.0 1.0 0.9 1.1 0.7 Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. 25 Detailed statistics on each question in the vignettes can be found in Appendix B. 12 Health prevention and promotion A central role of APEs is to engage in community health promotion and preventative health care. A complementary section of the medical vignettes tested APEs’ knowledge and engagement with the community, primarily concerning nutrition services and family planning. Nutrition APEs were asked to list all nutrition services they provided. Ninety-six percent of APEs provided at least one service, the most common being counseling on complementary feeding (80 percent) and exclusive breastfeeding (78 percent) (see Figure 8). APEs also revealed a satisfactory knowledge in these domains, with 88 percent correctly identifying the optimal duration of exclusive breastfeeding and 77 percent recognizing the correct approximate age range to begin complementary feeding. Other essential nutrition services were less prevalent, such as the distribution of Vitamin A and micronutrients in powder form (MNPs) and growth and weight monitoring of children. This could be partly due to issues hindering the procurement of scales and MNPs at the national level and a limited supply of vitamins in the APE kit (only 48 percent of APEs had Vitamin A+E in their drug kits at the time of verification.) APEs demonstrated poor knowledge regarding the ideal frequency of deworming (only 56 percent responded correctly), and only 55 percent reported offering deworming services. Figure 8: Percentage of APEs Who Offered Nutrition Services 80% 78% 59% 55% 40% 28% 19% 5% Counseling on Counseling on Community Deworming Distributing Growth Monitoring weight Distributing complementary exclusive counseling on children Vitamin A monitoring by age for micronutrients in feeding breastfeeding water, hygiene, children between powder form and sanitation 0 and 24 months Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Family planning Knowledge and practices related to family planning provisions indicated that substantial improvements are needed. While most APEs reported offering family planning methods (89 percent said they offered oral contraceptives [OCs], and 82 percent claimed to provide male condoms), the physical verification process of items in the kit showed a persistent lack of family planning methods. At the same time, 7 percent of APEs claimed to provide implants, which is not a method that is provided through community- based services. Additionally, 84 percent of APEs could not name any traditional methods of contraception, and a question regarding the differences between two commonly distributed OCs revealed that most APEs could not meaningfully differentiate them. 13 Medicines and supplements The physical verification of medicines and supplies was conducted with roughly half of the APEs in the sample (790). During this process, the enumerator asked the APE to open their drug kit and verified its contents, including the availability of key supplies, their quantity of each item, and their expiration date. This assessment revealed critical shortages in key supplies, such as family planning methods, antibiotics, and analgesics, as shown in table 4. At the time of verification, less than 50 percent of APEs had each of the three main methods of contraception, and only 10 percent had all three, posing severe limitations to women’s informed choice and uptake of their desired family planning and challenging the protection from sexually transmitted infections (STIs) that double- method use can provide. Similarly, the verification process revealed very low rates of antibiotics and analgesics, as well as low availability of important micronutrients. Availability was highest for rehydration salts (74 percent of surveyed APEs) and antimalaria treatment, where about 50 percent of APEs had some formulation of Coartem available. Table 4: Percentage of APEs with a Stock of Key Medicines and Supplements Percent of APEs 95 Percent Confidence Interval Family planning methods Oral contraceptives: Microgynon or Microlut 39 34–41 Injectables 104 mg/0.65 ml 29 26–32 Male condoms 35 32–39 Malaria diagnosis/treatment Rapid diagnostic test 62 58–65 Any combination of Artemeter+Lumefantrina (Coartem) 55 51–58 Artesunate 200 mg 17 14–20 Artesunate 50 mg 12 10–14 Micronutrients Iron 90 m+folic acid 1 mg 57 53–60 Vitamin A+E 48 45–52 Zinc 20 mg 44 40–47 Rehydration salts Oral rehydration salts (ORS) 74 71–77 Antibiotics, analgesics, and other Amoxicillin 125 mg 14 12–16 Amoxicillin 250 mg 16 14–19 Tetracycline 1% 37 34–41 Paracetamol 500 mg 44 41–48 Paracetamol 250 mg 34 31–38 Mebendazole 500 mg 44 40–47 Cetrimide 15%/Chlorhexidine di-glunato 1.5% 500 ml 27 24–30 Chlorhexidine gen 7.1% 3 2–5 The availability of medicines also differed among provinces (see Figure 9). For example, in Maputo Province, 66 percent of APEs had at least one method of female contraceptives (OC or injectables) in addition to male condoms. However, in Tete and Nampula, less than 10 percent did. Tete had the lowest availability of family planning methods. Availability of oral rehydration salts (ORS) was generally high, except in Niassa and Tete, where less than 50 percent of APEs had ORS. In four provinces— 14 Niassa, Nampula, Zambezia, and Sofala—less than 50 percent of APEs had any formulation of “Coartem.� Figure 9: Share of APEs Who Had Three Categories of Medicines at the Time of Verification 120 100 Percentage of APEs 80 60 40 20 0 Niassa Nampula Zambezia Tete Manica Sofala Inhambane Gaza Maputo Province Contraceptives (dual method) Coartem Oral Rehydration Salts Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Stockouts and medicine shortages have also been found in other recent studies, which have pointed to the need to address the different epidemiological needs of each province–given that the kit is standardized irrespective of the burden of disease/population demands of each province. Concerns related to the availability of medicines are also linked to the capacity of APEs to diagnose and treat patients correctly.26 A study currently underway by University Eduardo Mondlane (UEM) and the United Nations Children’s Fund (UNICEF) includes 94 APEs across four districts in Mozambique (Mossuril in Nampula, Angonia in the Central Region, Jangamo and Moamba in Inhambane, and Maputo Province, respectively). APEs were found to collect a new kit once every three months in three out of the four districts (except for Inhambane), which was also confirmed in the present study. However, the UEM/UNICEF study revealed difficulties from the APEs in understanding and filling out the stock management forms to be used. Throughout the visits, it was measured that paracetamol (250 g and 500 g) and amoxicillin were the most used medicines in the APE kit. Community engagement During the physical verification visit, APEs were also asked to show their registers of community engagement, consultations, and supervision received. 358 APEs (26 percent) did not have available reports. Answers were collected from the remaining 1,019 APEs. Community engagement includes APE visits to households, to foster promotive and preventative health or follow up on a treatment, as well as community health talks. On average, APEs had conducted 34 house visits over the previous month and given 27 talks over the previous three months (Figure 10). About 6 percent of APEs had not logged any community talks or household visits over the stated period. APEs logged, on average, 61 consultations over the previous months, for an average of 2 consultations per day. 26 UEM, UNICEF. October 2020. 15 Figure 10: Composite of Community Engagement of APEs by Province 80 72 68 Number of house visits + talks per month 70 60 55 51 50 46 40 40 34 28 30 20 10 10 0 Niassa Sofala Gaza Manica Tete Nampula Maputo Zambezia Inhambane Province Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. User satisfaction User satisfaction is also an important component of an analysis of the success of a community health program. The user survey interviewed 2,404 rural users over the phone regarding their experiences with APEs, attendance of APEs’ community health talks, and overall satisfaction with the service provided. Surveyed users skewed male (67 percent), with an average age of 38. One-half reported agriculture or fishing as their primary occupation, and the median monthly income was Mt 640–1,600. The average household size of the surveyed users was seven people, which included four children. Individuals were selected among lists provided by the APEs, which introduces a selection bias in the sample. Nevertheless, results suggest that rural communities rely on APEs to a large extent. When asked where they or someone in their household went the last time they felt sick, close to 80 percent said to an APE, while only 14 percent went to a health center and 5 percent to a hospital. Eighty-seven percent of users reported that their APE provided regular health talks, mainly on a weekly basis. Seventy-five percent of users reported attending the APE talks and went to an average of four talks over the past year. The most mentioned topic for the talks (Figure 11) was malaria (reported by 63 percent of users), followed by COVID-19 (50 percent) and water and sanitation (41 percent). The subjects of the APEs’ talks are primarily aligned with the topics in which APEs appear to be receiving the most training (85 percent received a refresher course on malaria versus 31 percent on HIV management). However, less than 50 percent of users reported that their last APE’s talk had included discussions of family planning, an important part of the APEs’ work. A large share of users also reported being satisfied with the care received by APEs. Most users were satisfied with APEs on several domains, namely APE availability and engagement and availability of medicines. The evidence gathered in the study does not fully support these perceptions, especially regarding the availability of medicines and supplements. It is important to note that over 50 percent of the survey respondents were male with an average age of 38. At the same time, family planning was one of the weakest areas for APEs, both in terms of availability of family planning methods, APE engagement (community talks), and knowledge. Moreover, the reliability of patient satisfaction with the quality of service is not established. A systematic review of 195 studies in developed countries found 16 little evidence of the reliability of patient satisfaction data.27 Other studies conducted in developing countries found similar biases in user responses on the quality of care.28 Figure 11: Topics Discussed by APE in the Last Talk (as reported by community members) 63% 50% 41% 36% 34% 18% 14% 9% 8% 6% 2% 2% Malaria COVID-19 Water and Diarrhea Family Nutrition Pregnant Newborn HIV Pneumonia/ Other Don't know sanitation planning women care acute respiratory infection Note: Percentage of respondents who reported their APE discussed each topic in last talk. Multiple options were allowed. Source: Quality of Care (QoC) survey of APE, 2020. Authors’ own calculations. Factors influencing APE performance Variable selection and model The framework of the analysis was informed by previous studies, which evaluated the impact and cost- effectiveness of APEs in Mozambique, as well as the factors that contributed to the success of the program. Receiving feedback and training from health authorities was found to increase APE motivation. Bowser et al. (2015) documented that adding APE salaries led to an efficiency gain of 56 percent in cost per output in the long run.29 Supervision is another key factor for the success of community health worker (CHW) programs. Ndima et al. (2015) found misalignments between policy and practice regarding skills, training, and supervision of APEs.30 For the purpose of this exercise, APE performance was defined as the ability of an APE to identify some of the most common clinical cases they encounter in their community health work. Essentially, the three hypothetical cases presented to the APEs were consistent with cases of malaria in children, malaria in adults, and diarrhea/respiratory infection in children. To be considered successful in identifying such cases, an APE had to diagnose at least two out of the three cases correctly. Hence, a dummy variable was created as the dependent variable, which takes on a value of 1 if an APE could correctly identify at least two of the three cases and a value of 0 otherwise. Based on a review of the relevant literature and the variables available in the data, several variables were identified a priori as impacting APE performance (see table 5). Given the binary nature of the dependent variable, a logistic regression model was chosen to analyze potential predictors of APE 27 J. Sitzia. 1999. How valid and reliable are patient satisfaction data? An analysis of 195 studies, International Journal for Quality in Health Care, Volume 11, Issue 4, August 1999, pages 319–328, https://doi.org/10.1093/intqhc/11.4.319. 28 See, for example: Glick, Peter. 2009. "How reliable are surveys of client satisfaction with healthcare services? Evidence from matched facility and household data in Madagascar." Social science & medicine 68.2: 368 –379. 29 Bowser, D., Okunogbe, A., Oliveras, E., Subramanian, L., and Morrill, T. 2015. A cost-effectiveness analysis of community health workers in Mozambique. Journal of primary care & community health, 6(4), 227–232. 30 Ndima, S. D., Sidat, M., Ormel, H., Kok, M. C., and Taegtmeyer, M. 2015. Supervision of community health workers in Mozambique: a qualitative study of factors influencing motivation and programme implementation. Human resources for health, 13(1), 1–10. 17 performance. Unlike linear regressions, logistic regressions do not assume normality and homoscedasticity. However, key assumptions still apply, namely linearity, independence of observations, and little to no collinearity among independent variables. Logistic regressions also require a relatively large sample size, which should increase when more independent variables are included in the model. Table 5: Independent Variables Thought to Impact QoC Given by APEs APE individual factors Community factors • Age • Location (province) • Sex • Distance from reference health facility • Education (time traveled to and from facility) • Years as an APE • Frequency of supervision (monthly supervision visits) • Ratio of supervisors to APEs • Training • Workload • Training in upSCALE Different linearity tests were performed for continuous variables, including visual examination of each variable plotted against their log odds, as well as the Box-Tidwell test. The Box-Tidwell test for linearity revealed a potential issue of nonlinearity with the APEs’ caseloads (number of patients seen by APE per month), and, as a result, the variable was removed. Explanatory variables were found to be independent and did not exhibit significant collinearity. The gender of an APE did not show a significant correlation with the dependent variable. Surprisingly, the frequency of training was also not significantly correlated with an APE’s performance, whether it was measured as the number of “refresher� courses the APE received or the number of times the APE received training over the previous three months. Hence, these two variables were excluded from the multivariate regression. No issues of collinearity among the independent variables were detected.31 A two-level model nesting APEs within provinces was employed. Results Univariate regressions were run with each variable. Age was regressed both as a continuous and categorical variable by constructing age brackets. In the univariate regressions, all variables had a positive effect on APE success, with different degrees of magnitude and significance. An older APE (over 35) had considerably greater odds of success, suggesting that experience builds knowledge. However, the number of years as an APE had quite a small effect. In the univariate regression, each additional year of experience led to an 8 percent greater chance of correctly diagnosing the clinical cases. Education had the greatest impact. Multivariate regressions included all the variables. Different specifications were considered for the multivariate logistic regression (table 6). In Model 1, the lower number of observations results from missing data in one variable—the number of supervision visits per month—due to many APEs not having their registers available for verifications. In Model 2, missing data points were predicted through multiple imputation. Multiple imputation is a general approach to deal with missing data, which allows for the uncertainty about the missing data by creating several different plausible imputed data sets and appropriately combining results obtained from each of them. It used a highly correlated variable to predict the missing values, which in this case were the number of supervision visits APEs reported to have received (as opposed to the verified registers of the number of supervision visits they actually received). After controlling for other variables, APEs’ years of experience as an APE, their training in upSCALE, and distance traveled to the nearest health facility lose significance. Education has the largest effect; having 31 Two variables, the age of an APE and the years of experience, had a correlation coefficient of 0.5. However, the literature suggests that correlation coefficients up to 0.7 are acceptable among independent variables. 18 completed primary education made an APE between 1.8 and 3.0 times more likely to diagnose at least two of the three cases. The number of supervision visits also revealed a significant effect. Each additional supervision visit per month could make an APE between 13 and 17 percent more likely to perform well. In contrast, being assigned to a supervisor with a higher load of APEs made APEs less likely to succeed in Model 2, which shows that an increase of one APE per supervisor could lead to a 3 percent drop in the likelihood of achieving the positive outcome. Table 6: Regression Analysis Results APE success=1 if APE correctly diagnosed at least 2 out of 3 cases Univariate regressions Model 1 Model 2 Odds ratio p-value Odds p-value Odds ratio p-value ratio Age 1.02*** 0 1.01 0.392 1.02* 0.050 19–25 (ref) 1 — 26–30 1.22 0.262 31–35 1.26 0.208 36–45 1.49* 0.031 46–55 1.83** 0.004 56–74 2.28** 0.003 Education Some primary (ref) 1 — 1 . 1 — Primary 2.47*** 0 1.79* 0.054 3.05*** 0 Basic 2.76*** 0 2.47** 0.007 4.78*** 0 Middle general 2.61*** 0 2.75** 0.007 4.15*** 0 Technical 1 — 0 (empty) 0 (omitted) Middle 2.95** 0.004 0 (empty) 1.87 0.158 Years as an APE 1.08** 0 1.02 0.128 1.02 0.204 Supervision visits per 1.13* 0.019 1.17* 0.011 1.14* 0.034 month APEs to supervisor 0.94*** 0 0.99 0.350 0.97* 0.083 ratio Trained in upSCALE 1.02* 0.018 1.01 0.159 1.01 0.593 Distance to facility 0.92* 0.012 1.01 0.547 0.99 0.186 Cons 0.79 0.695 Cons (province) 2.627 2.167 Number 935 1,280 Note: Model 2 uses multiple imputation to replace missing data in one of the variables (supervision visits per month). * = p < 0.05; ** = p < 0.01; *** = p < 0.001. CONCLUSIONS This cross-sectional quality of care (QoC) survey reveals critical deficiencies in the community-based services that are currently provided in Mozambique. Despite the relatively large areas and share of the population covered by APEs, the study found that they appeared actively engaged with the community through regular door-to-door visits and talks. Members of the community suffering from ailments largely resort to APEs before seeking care at a health facility, perhaps pointing to how they are perceived as trusted and accessible points of care at the community level. However, APE’s knowledge was found to be extremely deficient. The ability to identify basic and common illnesses appeared inadequate, with less than one in three APEs able to correctly diagnose cases of pneumonia and malaria in children and adults. Significant differences were observed among provinces in this regard and appeared worse in the Central provinces of Zambezia and Tete. APEs’ knowledge and dissemination of health promotion information related to contraception and nutrition are also limited. Weaknesses in APEs’ knowledge are compounded by systemic constraints in the supply chain of APE kits. Shortages limit the contribution that APEs can make to integrated community case management for key infections in Mozambique, such as malaria, given that less than 50 percent of APEs had any formulation of Coartem at the time of verification in the provinces of Niassa, Nampula, and Zambezia—three 19 provinces with the highest parasite prevalence. The low availability of contraceptives also impairs APEs’ role in addressing the existing unmet need for contraception in rural settings of Mozambique. Women’s sexual and reproductive health choices are limited, as only 10 percent of APEs had all three main methods of contraception (oral contraceptives [OCs], injectables, and male condoms). The fragilities in the program are not dissimilar from previous findings, which have highlighted shortages in medicines and variable implementation success.32, 33 The availability of medicines also differed among provinces, with southern provinces such as Maputo Province and Gaza having a better stock of critical medicines than Northern or Central provinces. Factors influencing APE performance were examined through regression analysis. Findings suggested that an APE’s level of education, followed by the strength of their supervision, contributes most to their knowledge, which is measured as the ability to correctly diagnose at least two of the three cases presented in the clinical vignettes. The study found that although all APEs have undergone initial training, refreshers have been delayed and have not benefited all equally, but, interestingly, APEs’ trainings were not significantly correlated with their ability to diagnose common ailments. Study findings could be used to improve the APE program through a combination of short- and long-term strategies. In the short term, it would be important to analyze in further detail the issues behind the supply chain that hinder the availability of medicines of APEs. Similarly, strengthening APE supervision, both in terms of frequency as well as the standardization of the supervision, appears paramount to the program’s success. This could be combined with on-the-job training to focus on common ailments that APEs ought to be able to identify and treat correctly and ensure the accuracy of the messages disseminated during health promotion activities. In the long term, recruitment criteria should consider the level of education of candidates, and the severe gaps in basic knowledge would also suggest that a revision of the curriculum and approach to training are needed. 32 Baltazar G. M., Chilundo, Julie L. Cliff, Alda R. E. Mariano, Daniela C. Rodríguez, and Asha George. 2015. Relaunch of the official community health worker programme in Mozambique: is there a sustainable basis for iCCM policy?, Health Policy and Planning, Volume 30, Issue suppl_2, December 2015, pages ii54–ii64. 33 UEM, UNICEF, Avaliação do sistema de abastecimento de medicamentos para os Agentes Polivalentes Elementares em quatro distritos de Moçambique no período de Fevereiro de 2019 a Agosto de 2020. October 2020. 20 Appendix A: Descriptive statistics of APEs and supervisors Descriptive statistics of APEs Number Percent Mean 95 Standar Percent d Confide deviatio nce n Interval CI Age 1,377 36 10.7 Gender 1,377 Male 832 60 58–63 Female 545 40 42–37 Median level of schooling 1,377 Primary Years as an APE34 1,375 4.6 5.5 Households under care 1,371 658 869 Average weekly hours worked in this role 1,376 17 10.08 Trainings in 3-month period 1,372 1 3.5 Descriptive statistics of APE supervisors Number Percent Mean 95 Percent Standard Confidence deviation Interval Age 320 31 6.6 Gender 320 Male 220 69 63–74 Female 100 31 37–26 Position/job title Nurse 94 29 Health technician 90 28 Director 57 18 Clinical Director 12 4 Public health doctor 10 3 General doctor 2 1 Health specialist 1 .3 Other 54 17 Years as an APE supervisor 320 2.4 2.7 APEs under supervision 320 5 3.6 Weekly hours worked in this role 320 5 4.7 34 At the time of data collection (2020). 21 Appendix B: Detailed responses to medical vignettes (in Portuguese) CASE 1: “Abel� (4-month-old boy with cough and diarrhea) Questions (diagnosis) APEs Number Percent 1.1.1 Duração da tosse 980 72 1.1.2 Pergunta se a criança teve febre 677 49 1.1.3 Pergunta se tinha algum sangue nas fezes da criança 338 25 1.1.4 Pergunta se a criança vomitou 399 29 1.1.5 Pergunta se criança teve convulsões 169 12 1.1.6 Conta as respirações da criança 266 19 1.1.7 Executa o teste rápido de sangue para a malária em crianças 475 35 1.1.8 Pergunta sobre a duração da diarreia da criança 683 50 1.1.9 Pergunta sobre a duração da febre da criança 402 29 1.1.10 Pergunta/verifica se criança consegue beber/ amamentar 280 20 1.1.11 Pergunta/verifica se a criança está com muito sono / inconsciente 121 9 1.1.12 Pergunta se a criança teve qualquer outro problema 324 24 1.1.13 Verifica se o peito da criança tem tiragem subcostal 134 10 1.2.1 Identifica a criança como tendo respiração rápida/pneumonia 496 37 1.2.2 Identifica a criança como tendo malária 233 17 1.2.3 Identifica a criança como tendo sinal de perigo / sintoma referência para 338 25 referência ou doença grave (qualquer) 1.2.4 Identifica a criança como tendo diarreia 444 33 1.2.5 Prescreve antibióticos; amoxicilina 400 30 1.2.6 Prescreve antimalárico; Coartem 171 13 1.2.7 Prescreve comprimidos de zinco 401 30 1.2.8 Prescreve SRO 543 40 1.2.9 Aconselha dose de Coartem—3 dias 120 9 1.2.10 Aconselha dose de amoxicilina—5 dias 221 16 1.2.11 Aconselha dose de SRO—quantas vezes necessárias 345 26 1.2.12 Aconselha dose de zinco—10 dias 199 15 1.2.13 Dá a primeira dose de amoxicilina 258 19 1.2.14 Dá a primeira dose de AL/Coartem 124 9 1.2.15 Dá a primeira dose de SRO 381 28 1.2.16 Dá a dose retal artesunato/plasmotrim 107 8 1.2.17 Dá a primeira dose de zinco 302 22 1.2.18 A criança não precisa de qualquer tratamento do APE 240 18 1.2.19 Refere à unidade de saúde 764 56 1.2.20 Escreva a nota de referência 724 53 1.2.21 Aconselha o uso de rede mosquiteira para criança 284 21 1.2.22 Aconselha alimentação contínua de liquidos para criança 280 21 1.2.23 Organiza uma visita de acompanhamento 464 34 1.2.24 Aconselha a devolver ou levar a criança a unidade sanitária caso agrave a 381 28 saúde 1.2.25 Regista a visita no livro de registo APE 658 48 1.2.26 Verifica o registo de vacinação da criança 202 15 22 CASE 2: “Deolinda� (10-month-old with a temperature; has not eaten in 3 days and cannot stay awake) APEs Number Percent 2.1.1 Identifica a criança como tendo respiração rápida/pneumonia 141 10 2.1.2 Identifica a criança como tendo malária 470 35 2.1.3 Identifica a criança como tendo sinal de perigo/sintoma de referência ou doença 750 56 grave (qualquer) 2.1.4 identifica a criança como tendo diarreia 150 11 2.1.5 Prescreve antibióticos; amoxicilina 162 12 2.1.6 Prescreve antimalárico; Coartem 258 19 2.1.7 Prescreve comprimidos de zinco 104 8 2.1.8 Prescreve ORS 131 10 2.1.9 Aconselha dose de AL/Coartem—3 dias 217 16 2.1.10 Aconselha dose de amoxicilina—5 dias 96 7 2.1.11 Aconselha dose de ORS—quantas vezes forem necessárias 104 8 2.1.12 Aconselha dose de zinco—10 dias 80 6 2.1.13 Dá a primeira dose de amoxicilina 100 7 2.1.14 Dá a primeira dose de AL/Coartem 184 14 2.1.15 Dá a dose de SRO 127 9 2.1.16 Da a primeira dose de retal artesunato/plasmotrim 139 10 2.1.17 Da a primeira dose de zinco 109 8 2.1.18 A Criança não precisa de qualquer tratamento do APE 227 17 2.1.19 Refere a criança à unidade sanitária 924 67 2.1.20 Escreve nota de transferência 926 67 2.1.21 Aconselha o uso de rede mosquiteira para criança 333 25 2.1.22 Aconselha alimentação continua e dar líquidos para criança 265 20 2.1.23 Organiza uma visita de acompanhamento 512 37 2.1.24 Aconselha a devolver ou levar a criança a unidade sanitária caso se agrave 368 27 2.1.25 Regista a visita no livro de registo APE 629 46 2.1.26 Verifique a saúde da criança ou o registo de vacinação 166 12 23 CASE 4: “Eduardo� (positive RDT) APEs Numbe Percent r 4.1.1 Identifica o paciente como tendo respiração rápida/pneumonia 95 7 4.1.2 identifica o paciente como tendo malária 1055 77 4.1.3 Identifica o paciente como tendo sinal de perigo/sintoma de referência 152 11 ou doença grave 4.1.4 identifica o paciente como tendo diarreia 105 8 4.1.5 Prescreve antibióticos; amoxicilina 182 13 4.1.6 Prescreve antimalárico; Coartem 845 62 4.1.7 Prescreve comprimidos de zinco 108 8 4.1.8 Prescreve ORS 104 8 4.1.9 Aconselha dose de AL/Coartem—3 dias 900 66 4.1.10 Aconselha dose de amoxicilina—5 dias 78 6 4.1.11 Aconselha dose de ORS—quantas vezes forem necessárias 94 7 4.1.12 Aconselha dose de zinco—10 dias 74 6 4.1.13 Dá a primeira dose de amoxicilina 91 7 4.1.14 Dá a primeira dose de AL/Coartem 538 39 4.1.15 Dá a primeira dose de SRO 108 8 4.1.16 Dá a dose de retal artesunato/plasmotrim 92 7 4.1.17 Dá a primeira dose de zinco 98 7 4.1.18 O Paciente não precisa de nenhum tratamento de APE 233 17 4.1.19 Refere a unidade sanitária 290 21 4.1.20 Escreve nota de referência 298 22 4.1.21 Aconselha o uso de rede mosquiteira 654 48 4.1.22 Regista a visita no livro de registro APE 600 43 4.1.23 Organiza uma visita de acompanhamento 513 38 4.1.24 Aconselha a voltar ou a ir para uma unidade sanitária caso se agrave 406 30 24 Appendix C: APE official medicine kit list in Portuguese and categorization used Item Category Tetraciclina 1% bisnagas de 5 g Topic agent Clorexidina em gel, 7, 1% Topic agent Paracetamol 500 mg Analgesics Paracetamol 250 mg Analgesics Amoxicilina 250 mg Antibiotic Mebendazol 500 mg Antiparasitic Cetrimida 15%/Clorexidina di-gluconato 1, 5%, 500 ml Antiseptic Adesivo (2,5 cm x 5 m) Consumable Algodão Hidrófilo, estéril, 500 g Consumable Compressa esterilizada (10 cm x 10 m) Consumable Ligadura de gaze (10 cm x 10 m) Consumable Envelopes de plástico para embalar comprimidos (60 x 80 x Consumable 0.025 mm) Sabonetes de 110 mg Consumable Caixa Incinerador Consumable Luvas de observação Consumable Preservativos Masculinos Contraceptives Levonor + Etinil (Microgynon) Contraceptives Levonorgestrel (Microlut) Contraceptives 104 mg/0.65 ml injectables Contraceptives Hexacloreto de Benzeno 600 mg/60 ml Fungicide Sal Ferroso 90 mg + �cido Fólico 1 mg Micronutrients/supplements Zinco 20 mg Micronutrients/supplements Amoxicilina 125 mg Micronutrients/supplements Vitamina A + E (Composta) Micronutrients/supplements SRO de baixa osmolaridade Rehydration salts Testes Rápidos de Malária (com lancetas e pipetas) Test kit Antimalarial treatment Arteméter + Lumefantrina 20 mg/120 mg (6 x 1) (tablets) Antimalarial treatment Arteméter + Lumefantrina 20 mg/120 mg (6 x 2) (tablets) Antimalarial treatment Arteméter + Lumefantrina 20 mg/120 mg (6 x 3) (tablets) Antimalarial treatment Arteméter + Lumefantrina 20 mg/120 mg (6 x 4) (tablets) Antimalarial treatment Artesunato 50 mg (tablets) Antimalarial treatment Artesunato 200 mg (tablets) 25 Appendix D: upSCALE in Inhambane and Zambezia Share of APEs who were trained on upSCALE app APE is trained on upSCALE Yes No Doesn’t know Zambezia 37.0% 61.69% 0.26% Inhambane 96.6% 3.45% 0 APEs’ use of upSCALE Zambezia Inhambane Alwa Some No Alwa Some No ys times ys times Over the past month, the phone on which you access 64.0 12.5 23% 32.0 10.7 56.0 upSCALE functioned normally % % % % % Over the past month, the wifi/mobile data on your 66.0 13.0 20% 28.6 10.7 59.5 phone was functional % % % % % Over the past month the upSCALE application functioned 67.0 8.0% 24% 23.8 15.5 59.5 normally % % % % Over the past month, the solar-powered phone charger 26.6 3.0% 69% 29.8 3.6% 66.7 was functional % % % Number of households APE registered on upSCALE 142 194 (mean) Share of APE supervisors who were trained on upSCALE APE supervisor trained on upSCALE Zambezia Inhambane Yes 53% 29.7% No 47% 70.3% Supervisors’ use of upSCALE Zambezia Inhambane Alw Som No Doe Alw Som No Doe ays etim sn’t ays etim sn’t es kno es kno w w Over the past month, the phone on which 26% 57% 17% 0 27% 9% 64% 0 you access upSCALE functioned normally Over the past month, the wifi/mobile data on 29% 60% 9% 0 36% 0 64% 0 your phone was functional Over the past month the upSCALE application 29% 49% 20% 0 27% 9% 64% 0 functioned normally Over the past month, the solar-powered 29% 26% 40% 5.7% 0 0 91% 9% phone charger was functional 26