PERFORMANCE EVALUATION OF PUBLIC HEALTH LABORATORIES IN KENYA DISCUSSION PAPER FEBRUARY 2018 Joel Lehmann Bernard Muture Martin Matu Miriam Schneidman PERFORMANCE EVALUATION OF PUBLIC HEALTH LABORATORIES IN KENYA Main Results and Future Opportunities Joel Lehmann, Bernard Muture, Martin Matu and Miriam Schneidman February 2018 Health, Nutrition and Population (HNP) Discussion Paper This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. For information regarding the HNP Discussion Paper Series, please contact the Editor, Martin Lutalo at mlutalo@worldbank.org or Erika Yanick at eyanick@worldbank.org. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. © 2018 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. ii Health, Nutrition and Population (HNP) Discussion Paper Performance Evaluation of Public Health Laboratories in Kenya Main Results and Future Opportunities Joel Lehmanna Bernard Mutureb Martin Matuc Miriam Schneidmand a Health Sector Research Consultant, Infospective Research Ltd, Nairobi, Kenya b Department of Preventive and Promotive Health, National Public Health Laboratory, Nairobi, Kenya c Senior Laboratory Specialist and Regional Project Coordinator, East, Central and Southern African Health Community, Arusha, Tanzania d Lead Health Specialist, Health, Nutrition and Population Global Practice, The World Bank, Washington, DC, USA This document was prepared under the East Africa Public Health Laboratory Networking Project (EAPHLNP), which is funded by the World Bank and coordinated by the East, Central, and Southern Africa Health Community (ECSA-HC). Abstract: This paper presents findings from a performance evaluation of laboratories in Kenya supported under the East Africa Public Health Laboratory Networking Project (EAPHLNP). The aim of the evaluation was to document progress and lessons learned, to enhance performance and foster sustainability. This paper is designed to stimulate an internal discussion for stakeholders involved in the provision of public medical laboratory services in Kenya at the county and national levels, with a particular focus on those involved with the EAPHLNP. The evaluation reviewed five different aspects of performance: test turnaround time, volume, accuracy, client satisfaction and scores achieved in peer-audits on the stepwise laboratory improvement process towards accreditation (SLIPTA). The evaluation also examined the perceived importance of the various project interventions, laboratory worker motivation, and the relationship of laboratory workers with their clinical counterparts. The assessment used quantitative information from routine service statistics, and qualitative data from key informant interviews that captured stakeholder perceptions, concerns, and insights to help understand and contextualize findings. The evaluation found an increase over time in the SLIPTA scores at the EAPHLN- supported laboratories, along with indications that client satisfaction and test accuracy improved. Most personnel employed by the project were absorbed as permanent Staff. There were no conclusive findings about laboratory performance in terms of turnaround time, test volumes, and effect on outbreak response. Three factors were identified that led to strengthened relationships between laboratory professionals and clinical Staff: (i) availability of new tests, greater confidence in laboratory results, and enhanced respect for laboratory workers; (ii) joint training activities between laboratory and clinical Staff; and (iii) an increased number of routine interactions. Given the rapidly devolved context in Kenya, the authors recommend that stakeholders share key findings with county governments to inform the design of strategies for continued funding of high-quality public health laboratory services; discuss and expand use of results-driven approaches, including promotion of more effective use of data and iii of accountability mechanisms; and expand joint learning initiatives that bring together clinicians, surveillance officers, researches and laboratory Staff. Keywords: Laboratory strengthening, diagnostics, quality improvement systems, accreditation, and laboratory performance evaluation. Disclaimer: The findings, interpretations and conclusions expressed in the paper are entirely those of the authors, and do not represent the views of the World Bank, its Executive Directors, or the countries they represent. Correspondence Details: Joel Lehmann, Infospective Research Ltd., Nairobi, Kenya, joel.lehmann@gmail.com, +254738989950. iv v Table of Contents ACKNOWLEDGMENTS .............................................................................................. IX EXECUTIVE SUMMARY.............................................................................................. X BACKGROUND ............................................................................................................. 12 STANDARDIZED PERFORMANCE EVALUATIONS ............................................................... 12 The Evaluation in Kenya ........................................................................................ 13 EVALUATION DESIGN ..................................................................................................... 14 STUDY LIMITATIONS ....................................................................................................... 14 INFLUENCE OF THE PROJECT ON LABORATORY PERFORMANCE .......... 16 BACKGROUND ................................................................................................................ 16 TURNAROUND TIME ........................................................................................................ 17 Quantitative findings on TAT................................................................................. 17 TEST VOLUMES ............................................................................................................... 18 Quantitative findings on test volumes.................................................................. 19 TEST ACCURACY (EXTERNAL QUALITY ASSESSMENTS)................................................... 20 Quantitative findings ............................................................................................... 20 CLIENT SATISFACTION .................................................................................................... 22 Quantitative data on client satisfaction................................................................ 22 Quantitative findings on client satisfaction.......................................................... 22 SLIPTA AUDITS ............................................................................................................. 27 Quantitative findings on SLIPTA performance ................................................... 27 SOURCE EAPHLN DISEASE SURVEILLANCE AND OUTBREAK RESPONSE ........................ 28 Factors that enabled the increased role of laboratories for surveillance and response ............................................................................................................... 28 Quantitative findings on disease surveillance and response .......................... 29 THE ROLE OF THE INDIVIDUAL EAPHLN PROJECT INTERVENTIONS .... 31 Source: Authors ...................................................................................................... 31 Equipment and Infrastructure ............................................................................... 31 SLMTA Training and SLIPTA audits.................................................................... 32 Operational Research (OR) .................................................................................. 33 Facility Improvement Funds .................................................................................. 34 Mentorship ............................................................................................................... 34 Other training activities .......................................................................................... 35 Hiring of additional personnel ............................................................................... 35 Source: ? .................................................................................................................. 37 Supervision .............................................................................................................. 37 LABORATORY WORKERS SKILLS AND MOTIVATION ................................... 38 New skills ................................................................................................................. 39 PROFESSIONAL RELATIONSHIPS AND LABORATORY PERFORMANCE .. 41 vi STAKEHOLDER CONCERNS ON SUSTAINABILITY .......................................... 43 RECOMMENDATIONS ................................................................................................ 45 APPENDIX A: EVALUATION METHODOLOGY................................................... 48 SAMPLE AND DATA COLLECTION .................................................................................... 48 Performance-related questions ............................................................................ 48 Data sources and sample ...................................................................................... 48 Qualitative method .................................................................................................. 49 Data Analysis ........................................................................................................... 49 APPENDIX B: DATA SOURCES AND DESCRIPTION .......................................... 50 Turnaround time ...................................................................................................... 50 Test volumes ........................................................................................................... 50 Test accuracy .......................................................................................................... 51 Disease surveillance and outbreak response .................................................... 52 APPENDIX C: DETAILS ON AVERAGE TAT RETRIEVED FROM LIMS ........ 43 vii ACRONYMS Continuous medical education CME DDSR Disease Surveillance and Response Unit in the Ministry of Health District Health Information System – Software used for national health DHIS2 information reporting EAPHLN East Africa Public Health Laboratory Networking Project EQA External quality assessment HuQAS Human Quality Assessment Services – a Kenyan not-for profit organization that offers proficiency testing for laboratories (PT) HMIS Hospital management information system ISO Prefix used by the International Organization for Standardization, from the Greek īsos, meaning equal KENAS Kenyan Accreditation Service (National accreditation body) LIMS Laboratory information management system MoH Ministry of Health NMRL National Microbiology Reference Laboratory NPHL National Public Health Laboratories (Kenya) OR Operations research PBF Performance-based financing PT Proficiency Testing (quality assessment for laboratory workers) SLIPTA Stepwise Laboratory Quality Improvement Process Towards Accreditation SLMTA Strengthening Laboratory Management Towards Accreditation SOP Standard operating procedures TAT Turnaround time VDRL Venereal Disease Research Laboratory test (blood test for syphilis) viii ACKNOWLEDGMENTS The authors would like to thank numerous individuals for their generous inputs and contributions. This includes experts and project stakeholders who have openly shared their experiences and views, and many of whom also shared data and reports that were accessible to them: Ben Kitole (Sub-county laboratory coordinator, Kilifi), Charity Muringo, (NPHL Quality Assurance Officer), Christabel Misiko, (EAPHLN Project Operations Officer), Daniel Langat (Head of Disease Surveillance & Response Unit), Daniel Wekesa (Medical Laboratory Technologist, Kitale), David Juma (Laboratory Manager, Machakos), David M. Kareko (DDSR Officer), Fridah Sirima (Laboratory Manager, Busia), Henry Ogaro (Laboratory Manager, Kitale), Joshua Rotich (EAPHLN Project Coordinator), Mamo Umuro (Head of NPHL), Mohammed Abbey (Laboratory Manager, Wajir), Nelson Akenga (NPHL ICT Officer), Solomon Gikundi (Laboratory Scientist), Thomas Gachuki (Head of NPHL Quality Assurance), as well as other laboratory managers and Staff, clinical workers who generously shared their time and insights. We are grateful for the expertise and technical support by Kelly Silva (Laboratory Consultant) and Wei Han (Statistician), and acknowledge the helpful contributions from peer reviewers, Robert Oelrichs (Senior Health Specialist, Health, Nutrition and Population Global Practice, World Bank) and Gandham N.V. Ramana (Lead Health Specialist & Program Leader, World Bank). The authors would also like to acknowledge the excellent support from Yvette Atkins in the production of the final report. The authors are grateful to the World Bank for publishing this report as an HNP Discussion Paper. ix EXECUTIVE SUMMARY The aim of this evaluation of the East Africa Public Health Laboratory Networking (EAPHLN) project in Kenya was to document progress and lessons learned and suggest improvements for the way forward. The evaluation provides information for counties that currently manage the EAPHLN-supported laboratories, and offers suggestions on how other counties might strengthen their laboratory and diagnostic capacity. The assessment used both quantitative and qualitative data to assess performance and get a fuller picture of achievements and challenges. The main findings and conclusions on the key research questions are summarized below. How have the EAPHLNP-supported laboratories performed over time? The analysis found a positive trend in the volume of GeneXpert tests, one of the key technologies rolled out under the project, but mixed results for other test volumes. The acquisition of modern equipment improved the turnaround time (TAT) for the diagnosis of TB and microbiology tests. For other tests, clinical workers reported that there was still room for improvement, and laboratory data showed mixed results and were generally not conclusive. Service quality was perceived to have improved, attributed to Staff training, new equipment, and newly recruited, more qualified personnel. External quality assessment (EQA) results found that the EAPHLN interventions had a positive impact on performance, but the analysis was limited by sample size. The satisfaction of clinical Staff and patients improved slightly, with greater confidence in test results, a broader menu of tests offered, and a more welcoming attitude of laboratory Staff. There was a strong overall improvement in the quality scores on the Stepwise Laboratory Improvement Process Towards Accreditation (SLIPTA) at all laboratories between 2011 to 2016. How has the preparedness and outbreak response capacity changed? Key informants noted that participating laboratories became more involved in disease surveillance and outbreak response activities, such as recent cholera outbreaks. The cross-border surveillance committees and the joint investigations were a novelty in Kenya, and were viewed as best practice internationally in terms of disease control efforts. These positive accounts notwithstanding, a review of the outbreak and rumor log, which only had a small number of entries in each county, did not show a clear effect of the project on outbreak response. How did laboratory worker skills and motivation change over time? Respondents agreed that the project had a positive influence on the skills and motivation of laboratory personnel. Training enabled Staff to conduct newer and more performant tests, and to offer a larger menu of tests. The automated equipment was greatly appreciated by laboratory personnel, and led to increased confidence in test results by clinicians. Joint training with clinical Staff contributed to improved professional relationships and greater involvement of laboratory personnel in clinical care. Coordinated efforts to improve SLIPTA scores also led to greater collaboration and reinforced mutual accountability. Three factors were viewed as key to enhancing motivation: (i) working in a conducive environment with state of the art equipment; (ii) benefitting from opportunities for individual growth and professional development; and (iii) interacting regularly with clinicians and hospital directors. x Has the relationship between laboratory and clinicians changed? The improved laboratory services and facilities were reported to have strengthened relationships between clinicians and laboratory technologists. Several factors contributed to strengthened relationships: (i) introduction of new tests and increased confidence in laboratory results; (ii) joint training activities and more interactions between laboratory and clinical Staff. What is the perceived role of individual EAPHLN interventions? Facility upgrading received the highest average rank, followed by the Strengthening Laboratory Management Towards Accreditation (SLMTA) training programme and the regular SLIPTA assessments. The order of preference varied between professionals, with laboratory technologists emphasizing the importance of laboratory improvement funds for day-to-day operations. Most participants valued the mentorship program, but underscored the importance of improving regularity of this hands-on training. Several interview participants emphasized that the qualifications, skills, and motivation of the EAPHLN-hired personnel were excellent. While most respondents did not cite the operational research as a priority intervention, at the Kitale laboratory both laboratory and clinical Staff noted the value of the that enteric microbiology training and research. What are the main concerns of key stakeholders regarding sustainability? Key bottlenecks that were viewed as a threat to continuity and the maintenance of the gains attained were related to: (i) lack of a regular supply of reagents; (ii) lack of equipment service contracts; (iii) delays in the delivery and installation of equipment and; (iv) concerns over Staffing. The devolution of health services, which came into effect during the implementation of the project, provided new opportunities but some challenges. Several respondents pointed to gaps in laboratory equipment, including lack of power backup, disruptions in the air conditioning, absence of back-up machines in case of breakdowns, and limited integration of the laboratory information management system with hospital information systems. Recommendations Three broad recommendations were derived from the assessment with details provided in the main report: • Provide information to county health authorities about the benefits of strong laboratory services, and urge them to allocate and earmark budgetary resources to consolidate investments and sustain results. • Broaden and deepen the results-driven environment established during the first phase of the EAPHLN project, with more effective use of data, and stronger accountability mechanisms, where end-user feedback plays an increased role, and performance based financing is used more systematically. • Continue to invest in joint learning initiatives that bring together clinicians, surveillance officers, research and laboratory Staff, and organize exchanges between different laboratory sites. xi BACKGROUND The East Africa Public Health Laboratory Networking (EAPHLN) project aims to establish a network of accessible, high quality public health laboratories in the East African Community member states (Kenya, Rwanda, Tanzania, Uganda and Burundi). 1 The operation addresses the historical neglect of public health laboratories in East Africa. The original US$63.66 million project (Kenya, Rwanda, Tanzania, Uganda) was approved by the Executive Board of the World Bank in 2010 with US$15 million approved for Burundi in 2012 and an Additional Financing of US$50 million approved in 2015 (Burundi, Kenya, Tanzania, Uganda). Each country serves as a center of excellence in a key thematic area, piloting innovations, and sharing experiences and lessons learned across the network. Kenya leads the thematic areas of operational research and disease surveillance; Uganda leads laboratory networking and accreditation; Tanzania is focused on training and capacity building; Rwanda leads on information- and communication technologies (ICT), performance-based financing (PBF) and multi-drug resistant tuberculosis (MDR-TB); and Burundi co-leads PBF with Rwanda. The East, Central, and Southern Africa Health Community (ECSA-HC), in collaboration with the East African Community (EAC), is coordinating activities at the regional level and supporting knowledge sharing. As part of the ongoing learning and evidence-based approach supported under the project, a series of evaluations are being conducted to assess performance and tease out lessons learned. An initial evaluation was conducted in Rwanda. This current evaluation focuses on the experience of the project-supported laboratories in Kenya. Subsequent evaluations are planned for the other countries. The results from these evaluations are expected to provide feedback to EAPHLN laboratories to improve their performance as well as distill innovative practices and lessons which may be of broader interest across the network and beyond. STANDARDIZED PERFORMANCE EVALUATIONS To understand what is working and what is not, it is important to systematically document and evaluate progress, and to tease out challenges and lessons learned in all countries. To design the evaluation methodology, a conceptual framework was developed. The framework (shows the logic of the EAPHLN intervention with its different elements, and ways in which the project may have brought change at the supported laboratories. While most of the intervention components were similar in all countries, the pace and intensity differed. For example, the time mentors spent on site at the laboratories varied widely between countries. Moreover, some interventions, like performance based payments were initially implemented only in Rwanda and Burundi. In the other countries, including Kenya, PBF schemes started more recently. The initial evaluation conducted in Rwanda in 2015, focused on the country’s experience with Performance Based Financing (PBF) tied to progress on the SLIPTA scores. The evaluation also assessed overall performance of the project-supported laboratories in Rwanda that benefited from various interventions (for example, facility upgrading, mentorship, human resources, capacity building, and improvement funds). 1 For more details on the project see following link: http://www.worldbank.org/en/results/2016/06/07/east-africa-public-health- laboratory-power-of-networking 12 Figure 1.1: EAPHLNP Intervention Conceptual Framework (adapted from Kumar et al., 20161) Source: developed by authors The Rwanda evaluation showed that it was not possible to statistically tease apart the effect of the different intervention components. Nonetheless, the evaluation generated a range of insights and lessons; observed that PBF mechanisms have a positive effect on performance for measured indicators, and that the relationship of laboratory workers with clinical Staff improved as confidence in laboratory results increased over time 2. Following the completion of the Rwanda evaluation, the other countries expressed interest in similar assessments to take stock of achievements and innovations, enhance opportunities for learning; and introduce improvements in the way laboratories are organized and services are delivered. The Kenya evaluation builds and expands on the methodology developed for the Rwanda evaluation, and continues to document innovations, challenges and lessons learned. Each evaluation is adapted to the country specific context to consider the uniqueness of each health system as well as adapt recommendations to ongoing health reforms in these countries. The Evaluation in Kenya The aim of the Kenya evaluation was to document progress and lessons learned to assist county governments to effectively assume responsibility for the management of the laboratories, and to enhance performance and foster sustainability of project-supported interventions. It aims to contribute to the discussion on how to improve diagnostic accuracy and the response to public health emergencies. The evaluation explores the influence of the EAPHLN-funded interventions on laboratory performance, and to better understand the mechanisms by which these effects were achieved. The specific research questions in the Kenya evaluation are listed below. Performance 1) How has the performance of EAPHLN-supported laboratories evolved over the past years, in terms of test volumes, turnaround time, client satisfaction and test accuracy? 2) How has the preparedness and response evolved in terms of disease surveillance and disease outbreak over the past years? 2 Kumar, Meghan; Lehmann, Joel Thomas; Rucogoza, Aniceth; Kayobotsi, Claver; Das, Ashis Kumar; Schneidman, Miriam. 2016. East Africa - Public health laboratory networking project: evaluation of performance-based financing for public health laboratories in Rwanda. Health, Nutrition, and Population (HNP) discussion paper. Washington, D.C.: World Bank Group. 13 Process 3) How did laboratory worker skills and motivation change over time, and how do these changes relate to laboratory performance? 4) Has the relationship between laboratory professionals and clinicians changed over time and if so, how did this contribute to improvements in laboratory performance? 5) What is the perceived role of the individual EAPHLN project interventions in achieving improvements? 6) What are the main concerns of key stakeholders at the county and national levels and how can these be jointly addressed? EVALUATION DESIGN The performance-related questions were answered using mixed methods, combining the use of administrative data available in the Kenyan health system with input from interviews with a purposive sample of 25 key informants, including laboratory managers, laboratory technologists, project administrators, clinical Staff, county health management team members and a hospital director. Where possible, EAPHLN laboratories were compared with non-EAPHLN laboratories in Kenya. The units of analysis were county/district laboratories or, in some cases, counties. The study population involved all 5 EAPHLN supported laboratories (Busia, Kitale, Machakos, Malindi and Wajir) and as comparison a sample of public sector laboratories where the required data were readily available and accessible. The process-related research questions were answered based on qualitative data gathered in the key informant interviews, which provided descriptive information based on the perspectives of the participants. This included examples of successes and shortcomings of the project, and recommendations or future directions. A more detailed description of the evaluation methodology and of data sources is contained in Appendix A and B, respectively. Drawing on the methodological work used for the Rwanda evaluation, five different aspects of performance were included in the evaluation: (i) Test turnaround time (TAT); (ii) Test volumes; (iii) Test accuracy as assessed in external quality assessments (EQA); (iv) Client satisfaction; and (v) Scores achieved on SLIPTA 3 audits. The performance assessment relied predominantly on quantitative information derived from hospital or country-wide records and routine data. In addition, the key informant interviews sought to understand perceived performance and to obtain additional insights to help understand and explain the findings. As part of the qualitative interviews, participants were asked to rank the eight main EAPHLN interventions. 4 STUDY LIMITATIONS Both the qualitative and the quantitative elements of the evaluation are subject to limitations. The availability and accessibility of high-quality routine time series for test volumes, TAT and client satisfaction was limited, as earlier information systems were paper-based and not all data are computerized. Moreover, the selection of comparison sites was often dependent on the availability of data, which introduced a certain selection bias, especially for the assessment of TAT. 5 Thus, some of the findings presented are approximative, representing the best available data and comparisons that could be obtained at the time, and within the scope of the evaluation. The qualitative data represent a range of different perspectives, of which some are based on a few interviews. The main limitation with key informant 3 Stepwise Laboratory Improvement Process Towards Accreditation. For details, see http://www.who.int/tb/laboratory/afro- slipta-checklist-guidance.pdf 4 They were asked which intervention they would select, if they were a decision-maker, if they could select one of them. After they selected the first intervention, they were asked which one they would select next, and so on until all eight interventions were ranked from 1 to 8. 5 Where sufficient data were available, the comparison sample was designed to only include laboratories providing services at the same level as the EAPHLN laboratories (county hospitals that were formerly known as Level 4 hospitals). 14 interviews is that they are subject to biases, as each participant gives an account of their own subjective experience, and the range of perspectives is limited by the number of participants included in the study (i.e. for example, there was one county disease surveillance officer and one hospital director in the sample). 15 INFLUENCE OF THE PROJECT ON LABORATORY PERFORMANCE BACKGROUND The EAPHLN-supported laboratories in Kenya received: i. Facility upgrades that included the construction of five new laboratory buildings and the renovation of the central public health laboratory, new information technologies, and modern laboratory equipment. ii. Facility improvement funds which supported community outreach activities and addressed gaps in the day-to-day operation of the laboratories. iii. Recruitment of additional human resources. iv. Training and capacity building, including enrollment in the Stepwise Laboratory Management Towards Accreditation (SLMTA) program, scholarships for Masters/PhD degrees, and Field Epidemiology training v. Mentorship support. vi. Assisted supervision. EAPHLN laboratories were supported to improve disease surveillance and laboratory confirmation of outbreaks. In addition, they participated in three operational research studies in the areas of tuberculosis diagnostics, malaria drug resistance, and enteric diseases, and in several cases conducted off shoot proposals. Table 1.1 shows the number of personnel that were hired under the project for the five EAPHLN-supported laboratories, as well as major equipment purchased as part of the project. As a result of the project, personnel increased by 40 percent at the EAPHLN- supported sites. Several additional Staff were hired and based at the national level: three laboratory technologists, two laboratory scientists, two ICT officers, an operations officer and a construction monitoring officer. Table 1.1: Staff Numbers and Equipment Received by the 5 EAPHLN Laboratories Through the Project 6 Personnel hired Equipment purchased Lab PCR Technolog Rotorge EAPHLN Site ist ICT Vitek Bactec ne GeneXpert Kitale 6 (11) 0 x x Busia 6 (16) 1 x x x Machakos 3 (15) 0 x x x x Malindi 6 (10) 0 x x x Wajir 6 (16) 1 x x x Total 27 2 4 1 5 5 6 Parentheses show the number of government-employed lab technologists at the beginning of the project before the addition of personnel. 16 Staff Since 2010, the EAPHLN project in Kenya trained a total of 2,460 health personnel, out of a target of 3,719. The trainings were mostly focused on skill improvement, and included mentorships, good clinical laboratory practice, microscopy, monitoring and evaluation, SLMTA, ISO 15189 accreditation and general management skills, among other themes. Table 2 shows the number of officers trained in each financial year since project inception. Table 1.2: Number of Officers Trained Year Officers trained Target achievement 2010/2011 300 33% 2011/2012 854 62% 2012/2013 780 75% 2013/2014 180 77% 2014/2015 177 276% 2015/2016 169 211% Total trained 2,460 66% In the following sections, the five dimensions of performance that were evaluated are discussed in detail, including a description of the data that were used and issues with data completeness and availability. This is followed by an assessment of feedback received on improvement in disease surveillance and outbreak response. TURNAROUND TIME According to the accounts given during the interviews, the acquisition of modern equipment, especially the GeneXpert and the VITEK machine 7 made a positive difference. For example, the introduction of the GeneXpert reduced TAT for TB drug sensitivity testing (Rifampicin resistance) from weeks to hours. According to clinical Staff, the TAT for GeneXpert improved over the course of the project with patients receiving results more quickly. The time used to carry out and report the results on the GeneXpert tests also improved over the years, according to key informants who noted that even though this is an automated technology, additional efforts are required in terms of processing results and providing them expediently to clinicians and patients. For other tests – including routine tests - clinicians at more than one of the EAHPLN facilities reported that the turnaround times often remain unacceptably long. This includes for example hemograms, or urgent tests like CSF chemistry or bilirubin. Delays were attributed by clinicians and laboratory Staff to high workloads. Quantitative findings on TAT The quantitative data suggest that EAPHLN laboratories performed similar or worse than the comparison laboratory in terms of turnaround time, which seems consistent with the feedback received through interviews with key informants. Comparison data were readily available from one hospital only, Nakuru county referral hospital, a provincial facility with high volume utilization and stronger human resources that may not be the best comparator for the satellite laboratories. 8 To account for potential systematic errors and to control for initial differences between laboratories, the trend in turnaround times was analysed (difference in monthly average), rather than average turnaround time itself. The average difference in turnaround time (percentage change) for consecutive months was calculated for each test, and compared between intervention and comparison laboratories. The sample size was too small to conduct regression analysis or meaningful tests of statistical significance, therefore the results reported are descriptive only. 7 VITEK is an automated microbial identification system that uses mass spectrometry technology. It can detect antibiotic resistance. 8 See Appendix B for a detailed description of the data for the various quantitative analyses. 17 In Table 1.3 below, a negative value indicates an improvement (reduction) in turnaround time over time. TAT in EAPHLN sites became generally longer over time for the tests analysed. When comparing the trend to the comparison site (Nakuru County Referral Hospital), EAPHLN laboratories showed a more positive trend for two tests (the pregnancy diagnostic test and sickle cell test), while for seven tests the trend looks more negative than at the comparison sites. For the remaining two tests reported, there was no comparison data; for one of them (malaria blood slides) the TAT improved, and for the other (stool ova and cysts) it became longer at the EAPHLN sites. While the data that were available for analysis was not sufficiently comprehensive, and had limitations in terms of accuracy with large variations and outliers, it does suggest that EAPHLN laboratories performed similarly or worse than the comparison laboratory. A plausible explanation of these findings is that increasing test volumes and the additional efforts required to improve the quality of services increased the workload of Staff to an extent that it affected turnaround times (see page 34 on stakeholders concerns and sustainability). Table 1.2: Average Monthly Growth Rate of Test Turnaround Time, May 2015 - June 2016 Average monthly change of TAT (% change between months at t and t+1) Number of trend observations between EAPHLN Comparison months from Laboratories Laboratory April 2015 to June 2016 EAPHLN Comparison Blood Slide for Malaria Parasite -6.1 - 25 0 Erythrocyte Sedimentation Rate +28.8 -0.4 15 13 Fasting blood sugar +35.9 +22.8 9 8 H Pylori +5.9 -7.0 14 9 Hb +1.7 +1.8 21 12 Pregnancy Diagnostic Test -5.7 -0.5 22 13 Rheumatoid Factor +36.0 +2.1 21 13 Random blood sugar +17.3 +10.3 25 10 Sickle Cell Test -15.9 -0.3 6 12 Stool Ova and Cysts +10.3 - 22 0 Urinalysis -0.7 -3.1 23 12 Source: LIMS TEST VOLUMES In the interviews, laboratory managers pointed out that the number of tests performed had increased, both in terms of the range of tests offered, and the volume of tests performed. The estimates of the perceived increase in test volumes varied across interviewees, ranging from “around 10 percent more” to a “threefold increase.” The main factor to which they attributed increased test quantity was the increasing demand from clinicians and patients, which they saw as being driven by the broader test menu percent , greater trust in the results by health workers, and improved communications. In Kitale, a laboratory newsletter/circular now updates Staff about new tests, new regulations, as well as alerts in case of stock-outs or breakdowns, which helps to better manage demand. Staff at the Kitale laboratory reported that they also experienced challenges when trying to strike a balance between efficient management of specialized equipment that require batch testing on specific days, and timely testing of patients in need who often travel long distances to access care. In Wajir, the overall demand for health services remains relatively low, according to the laboratory manager; however, the increase in Staffing (with new 24-hour shifts) and the rise in demand for testing, rather than being satisfied with empirical management, led to an increase in test volumes. 18 QUANTITATIVE FINDINGS ON TEST VOLUMES Test volume data were readily available in the national Health Information System (HIS) database from three of the five EAPHLN laboratories 9. Thirteen comparison sites were chosen purposively to represent comparable counties and settings 10. Growth rates were analysed rather than absolute volumes, to control for initial differences. After removing outliers, the average difference for any two subsequent quarters was calculated for each test, and compared between intervention and comparison laboratories. The sample sizes were too small to conduct regression analysis or meaningful tests of statistical significance, therefore the results reported are primarily descriptive. In table 4 below, a positive value indicates an improvement (increase) in test volume. The findings need to be interpreted with caution, as the large observed variation between quarters in test volumes might be a sign of poor record-keeping. Table 1.4 shows that for 2 out of 7 tests (syphilis blood test (VDRL) and urine microscopy), the EAPHLN sites showed a higher average growth rate than the comparison sites. For two tests, the growth rates are similar (CD4 and full blood count), while for three tests, the comparison sites have larger growth rates. In conclusion, the quantitative data show mixed results in terms of test volumes but should not be overly interpreted given the relatively small sample sizes and missing entries. A possible explanation for the reduction in the volume of malaria tests in EAPHLN sites, if the records are an accurate reflection of number of tests ordered, could be a decreasing number of suspected malaria cases over time. Table 1.3: Comparison of Change in Test Volumes, Q1 2015 – Q2 2016 Average quarterly growth-rate of test volume, Q1 2015 - Q2 2016 (% change) Total number EAPHLN Comparison of Laboratories Laboratories observations CD4 +10.8 +9.3 57 Full blood count +19.0 +26.2 54 TB sputum smears -7.7 +10.1 59 VDRL +54.9 +25.8 56 Malaria BS (5 years and above) -32.6 +20.6 54 Malaria BS (under five years) +5.1 +16.5 55 Urine microscopy +51.8 +12.4 63 (Source: DHIS2) In addition to the data accessed through the HIS data base, a dedicated system for the GeneXpert machines provided information on the volume of tests carried out with this specialized equipment. The drop in the number of TB sputum smears seen Table 4 is likely a result of the increasing use of the novel GeneXpert technology which diagnoses TB faster and more accurately, with an increase of +268% in EAPHLN laboratories over the life of the project (+263% in Kenya overall). Malindi was the first laboratory among the EAPHLN sites to get the machine in early 2012, followed by Machakos and Kitale one year later, and more recently by Busia and Wajir. Malindi tested a relatively consistent number of around 100 samples per quarter over the years, while most other laboratories had larger fluctuations. As seen in Figure 2 below, since the inception of the project there was a generally increasing trend in test volumes (in spite of intermittent fluctuations) in utilization of the GeneXpert technology which was one of the main innovations supported under the project. The project also procured Vitek 2 machines for the identification of culture isolates and antibiotic 9 Key informants attributed the fact that two of the EAPHLN laboratories did not submit data to the national HIS database, at the assessed time, to the devolution of the health system. 10 EAPHLN laboratories included: Kitale, Busia and Wajir. Comparison laboratories included: Homa Bay, Kericho, Kisii Hospital, Kisumu County Hospital, Jaramogi Oginga Odinga Hospital, Mandera, Olkalou, Nyeri, Samburu, Hola, Siaya, Alupe, Garissa, Marsabit. 19 susceptibility testing. Data was available from three of the supported laboratories, the National Microbiology Reference Laboratory (NMRL), Kitale and Machakos. According to these data, a total of 1,429 isolates were positively identified between 2013 – 2016, of which 700 in the Kitale laboratory. Figure 1.2: GeneXpert Test Volumes Development Over Time Source? TEST ACCURACY (EXTERNAL QUALITY ASSESSMENTS) Subjectively, those interviewed for the evaluation perceived that the laboratories had been able to improve both the quality and accuracy of testing. The improvements were attributed to Staff training and change in Staff attitudes with greater attention to quality; new equipment and a greater trend towards automation; additional Staff who were often more qualified; and in some cases, due to the laboratory having the finances to buy sufficient reagents to run control tests, following quality protocols. Respondents at the Kitale laboratory noted that improvements in test accuracy resulted in higher demand from clinicians which in turn enhanced the reputation of the laboratory to deliver high quality results. In spite of increased Staffing levels, shortage of human resources was still cited as a threat to quality. Quantitative findings The Human Quality Assessment Service (HUQAS) 11 database of EQA results contained 5 categories of tests evaluated, including Chemistry/Immunoassay, Haematology, Lymphocyte Immunophenotyping, Malaria tests and Myobacterium Acid Fast Stain (TB smear). The sample size was large enough to run tests of statistical significance, and the data quality and evidence on EQA performance is therefore solid. The percentage of acceptable test results was compared between intervention and non-intervention laboratories separately for each test event, as well as for each type of test (across test events). Hospitals that were formerly called Provincial General Hospitals (Level 5) and reference laboratories were excluded from the analysis. Tests that were reported as “not evaluated” were excluded from the analysis 12. 11 HUQAS is a Kenyan not-for profit organization that offers proficiency testing (EQA) for laboratories. At the time of the evaluation, results were delivered through an online portal to NPHL, and could be exported as PDF files. The format was not optimized for longitudinal and comparative data analysis. For additional details on the EQA process, see appendix. 12 Appendix D contains an analysis with the inclusion of these cases. 20 In Haematology, EAPHLN sites performed better than comparison sites, and in the April 2014 and the July 2015 test events, EAPHLN sites performed better overall. In the July 2014 test event overall, the comparison sites performed better. These differences are statistically significant. For the remaining tests (chemistry/immunoassay, lymphocyte Immunophenotyping, malaria and TB smears), and for the April and November 2015 tests events, the performance between intervention and comparison sites had no statistically significant difference. The findings suggest that the EAPHLN intervention may have had a positive impact on performance in external quality assessments, especially if non-participation in planned EQA tests (due to, among other things, equipment downtime or reagent stock out) are excluded from the analysis. EAPHLN laboratories performed particularly well in the last two test events reported in the HUQAS network analysis database, achieving 100 percent acceptable results in November 2015. Table 1.4 Comparison of EQA Performance in 5 Tests Events, 2014 -2015 Source: HuQAS database 21 CLIENT SATISFACTION According to interview participants, efforts undertaken to improve client satisfaction included the establishment of a complaints register, client satisfaction surveys, an innovative SMS channel for client feedback in the case of Kitale, and a transparent complaint box that encourages faster response to feedback on concerns over turnaround times and test menus. Based on those interviewed on site, the client satisfaction of clinical Staff and patients improved over time, although not dramatically. The improved confidence in test results, the broader menu, and a more welcoming attitude of laboratory Staff were cited as key factors that contributed to greater satisfaction among clinicians. According to key informants, the large inside waiting area (protected from the weather) made an important difference in Machakos. Conversely, interviewees noted that there are still complaints about turnaround times at the three laboratories visited. Among clinical Staff, reagent stock-outs, equipment breakdowns and the perennial problem of under-Staffing in the laboratories were cited as having a negative impact on clinician satisfaction. Quantitative data on client satisfaction Laboratory managers shared reports and questionnaires from the client satisfaction surveys carried out in their laboratories during the EAPHLN project period. The sites which shared these details were Malindi, Kitale, Busia and Wajir. The surveys were developed independently by each site and were therefore not comparable. The data collection method (face-to-face vs. self-completion 13), sampling method, sample sizes, data entry and data analysis varied as well. The questionnaire items (often satisfaction scaled) and the sampling procedures require validation and strengthening before the data can be qualified as strong evidence. Two out of the four laboratories that provided questionnaires and reports on client satisfaction had written one report each on client satisfaction during the EAPHLN program; two (Malindi and Wajir) reported two data collection events. For the evaluation, all questionnaires, reports and raw data sets were reviewed and the methods summarized, together with highlights of the findings (Tables 4 and 5 below). Quantitative findings on client satisfaction Across the sites, patients and clinicians associated the highest satisfaction levels with laboratory Staff friendliness and attitude, and lowest level of satisfaction with turnaround times, which appears consistent with the data and information discussed in previous sections. For other areas such as the sample collection process, communication of results, and overall satisfaction, about two thirds of the respondents indicated satisfaction levels between average, good and excellent. Malindi reported a considerable drop in waiting times between 2014 and 2015, and an overall positive trend in client satisfaction; Wajir also reported a positive trend with an increase of 20 percentage points or more on satisfaction levels. An area that the laboratories can still improve on, based on the comparatively high percentage of dissatisfied clinicians reported in Kitale and Busia, is the “communication of rejected samples to ward/ clinic”. The strength of the methodology used for the client satisfaction surveys (questionnaire items, sampling, analysis and reporting) varied between sites. The use of validated standardized items would allow for comparability; regular surveys would allow to observe 13 The Busia laboratory had a questionnaire for self-administration; Malindi and Kitale had the questionnaire administered by research assistants who were not employees of the health facility, and in Wajir the method of administration is unspecified. 22 trends, and an intuitive, action-oriented display of results (reporting) would likely increase the impact of findings 14. 14 At the request of the countries participating in the EAPHLN project, the ECSA-HC has now produced a standardized questionnaire for tracking client satisfaction. 23 Table 1.6: Summary of Client Satisfaction Surveys Reports overview (1/2) Sample size Data entry Number of Sampling Patients Clinicians and Report highlights: Patients Report highlights: Clinicians reports methodology storage - For TAT, clinicians were most satisfied with - Very high satisfaction levels with microbiology, and least satisfied with CD4. laboratory Staff friendliness, cleanliness - Clinicians seemed satisfied with the and attitude. explanation and interpretation of results, but Data - TAT shows relatively poor rating, with less so with communication regarding rejected available 36% of clients rating it as poor, 28% as Random samples in Epi Info, average, and about 26% good and 9% sampling, - More than 50% related courtesy of laboratory 1 each for which was excellent. based on Staff as excellent or good, and less than 20% Kitale clinicians & 249 100 hard to - About 45% were satisfied with laboratory thought it was below average (except on phone, patients open and affordability, and the remaining 55% rated it register for where 1 in 4 clinicians was dissatisfied) utilize by as average or poor patients - there where multiple verbatim comments on Laboratory - for most other items (e.g. process of issues that are "so urgent that they cannot be manager sample collection, responses to inquiries, ignored" - which included improvement of as well as overall satisfaction), two out of communication, improve on loss of GeneXpert three clients responded "good" or samples, improve on speed, and expansion of "excellent". the test menu - One in 3 clients was dissatisfied with - For TAT, clinicians seem most satisfied with speed, 1 in 4 thought the speed was parasitology, and least satisfied with excellent, with the remaining being satisfied microbiology - Friendliness of laboratory Staff was rated - Communication on rejected samples & critical Data high among patients results should be improved; communication on 1 each for available - The most commonly mentioned areas to stock-outs, breakdowns and new tests is better Busia clinicians & 59 30 Not reported in MS improve on was speed, followed by the - Good satisfaction with courtesy and test patients Word statement that "everything" should be reliability report only improved (almost one out of 5 respondents) Less satisfaction with information that - 45% answered that they had recommended accompanies results and sample transport Busia to a friend; 80% answered that the - Most other areas have an apparently 'average' overall services were satisfactory or very satisfaction level satisfactory Source: Laboratory Managers Table continued on next page 24 Reports overview (2/2) Number of Sampling Data entry and Report highlights: Patients Clinicians Report highlights: Patients reports methodology storage Clinicians - The report of 2016 compares findings of 2014 and 2015 - An immense improvement of waiting times is reported between 2014 (60% longer than 30 min) and 2015 (5% Data are available longer than 30 min). in an SPSS file, - Patient satisfaction with laboratory Staff making them which is hard to 2 for Random feel comfortable appears to have declined deteriorated open and utilize by patients sampling; slightly between 2014 and 2015, but their satisfaction No satisfaction survey Malindi 158, 299 n/a Laboratory (2014, Patient exit with Staff attitude has improved among clinicians available manager. SPSS 2016) interviews - The overall patient satisfaction with services has output (tables) in improved MS Excel was also - The analysis includes some bivariate analysis and the available. report found a relationship between overall satisfaction and satisfaction with waiting times and other questionnaire items. 2 periods Data not available The poster does not clarify whether the reported results are on patients, clinicians, or reported in digital format an aggregation of both. The bar-chart shows an increase of 20-30 percentage points Wajir Not reported Not reported (2013, other than the in satisfaction levels across client management, results reliability, TAT and overall 2015) report satisfaction. 25 Table 1.7: Description of Methods Used to Assess Client Satisfaction Patient questionnaire Health worker questionnaire Analysis and reporting Comments Similar instruments in Kitale and Busia Patients 15 questions in total - Percentages for each response 11 scale items, using a 5-point scale category; in one analysis 'excellent' and The scaling for clinicians is labelled as follows: Excellent - above 21 scale items, rated on a 5-point scale 'good' were combined somewhat hard to understand 90%, Good - 80%, Average - 70%, labelled as follows: Kitale Clinicians and discontinuous (no scores Poor - Below 60%; Three yes/no - Percentage responses for each of the between 60-70%). This questions and one single-response Excellent (>90%), Good (>80%), Average positive response categories (i.e. >70%); applies to Busia as well. item with three answers (is the service (>70%), Below average (<60%) verbatim report of all open-ended better/same/worse as before? responses. In Kitale only, there were two additional Patients open-ended items: - A 3-point scale with two 9 questions in total - Percentage for each response category. positive and one negative Six scale items, using a 3-point scale Open ended responses were categorized, In your own opinion, what is the major items may cause a bias Busia (Unsatisfactory, satisfactory, excellent) and percentages reported problem in laboratory services? - The Clinicians satisfaction Two yes/no items, and one open- Clinicians What is so urgent that it cannot be ignored report is hard to interpret with ended item (areas for improvement) - Absolute numbers for each response to improve laboratory services? absolute numbers category 19 questions in total (+ age & gender) 11 yes/no items; Three scale items, - Percentages for the various response using a 4-point scale (Very good, good, items, comparison between 2014 and An analysis by age and fair, poor). Three single-response No health worker survey instrument or Malindi 2015 to understand the trend. Bivariate gender might have given items with pre-defined waiting time report available at the time of the evaluation analysis to understand relationship additional insights. categories & number of times pricked between different questionnaire items. for blood collection and two open- ended questions (compliments/complaints) - At the time of the evaluation, there was a 12 questions in total, (+ a field for 'other poster available that showed percentages 9 questions in total, (+ a field for comments') for four areas: Overall satisfaction, TAT, 'other comments') 10 scale items, using a 5-point scale Results Reliability, Client management. All of them scale items, using a 5- labelled as: Excellent (100%), Good (75%), Wajir Two percentages are shown, one 2013, point scale labelled as: Excellent Average (50%), Bad (25%), Poor (5%) and one 2015. From the poster, it is not (100%), Good (75%), Average (50%), Two open-ended questions about areas possible to tell how the scale from the Bad (25%), Poor (5%) the laboratory has improved, and areas it questionnaires was analysed to yield the should improve. percentages. Source: Laboratory Managers 26 SLIPTA AUDITS Countries in East Africa adopted the WHO-AFRO Stepwise Laboratory (Quality) Improvement Process Towards Accreditation (SLIPTA) as an approach to strengthen laboratory quality management systems. SLIPTA is a framework for improving quality of public health laboratories to achieve the ISO 15189 standards. Regular laboratory assessments provide information on the functionality of laboratory, capacity for diagnosis and surveillance of various diseases. The assessment employs a standardized checklist with a set of questions to assess the functionality and compliance of the laboratories to various quality systems elements 15. The process identifies weaknesses in the quality of laboratory services for disease detection and enables the development of an action plan to strengthen quality management systems. The laboratories are then rated/classified depending on their aggregate scores in a five-star scale from “0 Stars” (<55 percent score) to “5 Stars” (>95 percent score). A unique regional peer audit mechanism was introduced as part of EAPHLNP, whereby certified assessors trained by the African Society for Laboratory Medicine (ASLM) conduct annual peer assessments of laboratories in neighbouring countries. In this model, trained and certified laboratory auditors from one country conduct exchange assessments of the satellite laboratories in the neighbouring countries within the network. This helps to promote knowledge sharing, objectivity and transparency, while generating cost efficiencies, to the extent that these are carried out by regional experts rather than international consultants. Laboratory managers and Staff reported that initially, after the Stepwise Laboratory Management Toward Accreditation (SLMTA) training was conducted at the EAPHLN sites in May 2012, the change in practice that was required and the establishment of regular SLIPTA audits was met with some resistance among laboratory Staff. Ultimately, it led to a positive change in attitudes, and according to several respondents, quality now plays an important role in the mindset of laboratory Staff. According to one of the project administrators, the improvement at some sites was particularly noteworthy. For example, at the Machakos laboratory, which scored lowest in the first SLIPTA audit, a concerted effort was made to turn the situation around. Malindi improved documentation, but experienced some fluctuations in terms of SLIPTA scores in subsequent annual peer assessments. Interview participants both among project administrators in Nairobi and at one of the EAPHLN sites mentioned that the mentoring approach used in Kenya was not optimal for sustainable change: mentors typically only travelled to laboratories a few weeks prior to the peer audit in anticipation of the assessment. The same participants suggested that a more rigorous approach with longer time spent on site during the year would have led to more durable improvements. The change of practice at the laboratories to comply with the SLIPTA requirements were also noted by people outside of the laboratories: clinical Staff cited stricter access rules, and a county director of health was aware of improved protocols and procedures introduced at the laboratory. The SLMTA training, combined with the SLIPTA assessments, was perceived to be a ‘game-changer’ by many of the interview participants, from top NPHL leadership to the manager of the remote laboratories. Quantitative findings on SLIPTA performance A review of the stars and scores obtained by the different laboratories from project inception until the latest assessment shows that there was a strong overall improvement at all laboratories between 2011 to 2016. While some laboratories faced intermittent challenges in sustaining improvements, the difference between the initial assessment and the last evaluation was at least two additional stars at all sites, and it included an outstanding performance of the central microbiology laboratory (CML) which climbed from zero to 5 stars, and was recommended by the Kenyan Accreditation Service (KENAS) for 15Documents and Records, Management Reviews, Organization and Personnel, Client Management and Customer Service, Equipment, Internal Audit, Purchasing and Inventory, Process Control and Internal and External Quality Audit, Information Management, Corrective Action, Occurrence/Incident Management and Process Improvement and Safety. 27 accreditation, after an independent assessment. In Table 1.8 below, a green shading represents a gain of one or more star levels, while a red shading shows a loss in star levels. Table 1.8: SLIPTA Stars Achieved in EAPHLN Peer Assessments (Source: EAPHLN) SOURCE EAPHLN DISEASE SURVEILLANCE AND OUTBREAK RESPONSE According to the accounts of interview participants, laboratories have become more involved in disease surveillance and outbreak response activities as a result of the EAPHLN project. They now play an active role in the county-level multi-disciplinary surveillance and response teams, as well as in cross-county surveillance activities. Some participants pointed out that several of the EAPHLN laboratories serve as hubs that cover surrounding counties for testing. A respondent in Nairobi noted that epidemiologists have started relying on laboratory tests during outbreaks in the past few years, while in earlier times drugs were administered in outbreak areas relatively indiscriminately. Laboratories have also become more likely to be the place where outbreaks are first identified; recent cases of cholera outbreaks were mentioned as examples by multiple participants. One interview respondent mentioned that as a result of greater capacity, the labs have increasingly become a “go-to” place in situations of a sudden disease outbreak; a recent food poisoning event in a school was mentioned as an example. One of the laboratories was asked to develop a budget for outbreak readiness when the El Niño climate phenomenon (and potential flooding) was expected. Cross-border multidisciplinary committees comprised of disease surveillance and public health experts, laboratory experts, animal health officers, wildlife and immigration officials were established in eight hot-spot disease transmission areas, with the task of developing joint plans. Several outbreaks in cross-border areas were subsequently investigated jointly by a Regional Rapid Response team (for example, the 2012 Marburg and Ebola outbreaks in Uganda; the 2012 Cholera outbreaks between Kenya and Uganda in the Mbale region, a 2015 Cholera outbreak at the border of Burundi and Tanzania and most recently the 2017 Marburg outbreak at the border of Uganda and Kenya). The cross-border surveillance meetings organized through the project were a novelty in Kenya, and are viewed as best practice internationally in terms of disease control efforts. In Kenya, this practice was noted with interest by parliamentarians who asked the NPHL for a presentation on the collaboration. Factors that enabled the increased role of laboratories for surveillance and response Key informants stated that an important factor in the increased participation of laboratories in disease surveillance and response activities was the opportunity for laboratory professionals to collaborate with other specialists within the county, the sub-national region, and across national borders. Laboratory professionals became active members of surveillance and response teams. On the national level, the MoH Disease Surveillance and Response Unit (DDSR) now collaborates more closely with the National Microbiology Laboratory, a trend which was driven by the improved availability of reagents. This enabled the laboratory to be ready to perform tests when necessary. Another important factor was 28 that the laboratory improvement plan funds were sometimes used to bridge gaps for routine surveillance activities; specifically, to pay for operating costs. In Wajir County, devolution brought a positive change in that it enabled the hospital and laboratory team to respond more quickly and efficiently, even when the new laboratory building was still under construction. An interview participant noted that the operations research activities may also have contributed to the quick cholera response, through training provided to Wajir laboratory personnel on macroscopic appearance of cholera stool, as part of the OR activity, and the networking between the Wajir laboratory and the KEMRI enteric laboratory. Some interviewees reported that in the year prior to the evaluation, the responsiveness of the laboratories (in particular the national team) had declined. This was a result of the control layers that were introduced to strengthen financial management and governance practices. Likewise, at the county level the availability of funds slowed down with the devolution of health services and the conclusion of the first EAPHLN phase, which resulted in uncertainty on whether the same processes could be maintained for surveillance and outbreak preparedness. Quantitative findings on disease surveillance and response In a review of available data from the Disease Surveillance and Response Unit (DDSR) at the Ministry of Health (MoH), the average reporting rates for 12 months (in 2015) in counties with EAPHLN laboratories were compared with six purposively selected comparison counties (West Pokot, Kwale, Kitui, Makueni, Mandera and Marsabit). The goal was to assess whether the disease surveillance strengthening efforts under the EAPHLN facilities resulted in an improved facility reporting rate in comparison to non-intervention counties. As shown in table 6 (below), the reporting rates in the EAPHLN-supported counties were better than for the comparison counties. Due to the small sample size, the difference is not statistically significant. In addition, the reported number of days between the detection of a first suspicious case in a health facility until the date concrete interventions began was calculated, based on data recorded in the national “Disease Outbreak and Rumour Log” maintained by DDSR. Twenty- nine outbreaks in Nairobi were excluded from the analysis, assuming that the situation in the capital city is not comparable to that in the counties. The average number of days between detection and intervention was longer in EAPHLN intervention counties. Finally, the proportion of outbreaks that were confirmed or ruled out, vs. those that remained unknown, was compared between counties with an EAPHLN-supported laboratory, and a set of comparable counties that also had occurrences of outbreaks or rumours during the project period. The proportion of confirmed or ruled-out cases was somewhat higher in the counties that did not have an EAPHLN-supported laboratory (76 percent vs. 63 percent, based on a very small sample). These findings are based on very few observations, and do not consider that EAPHLN laboratories also supported surrounding counties and are located in hot spot, cross border areas with more frequent outbreaks; it is therefore merely indicative that there isn’t a project effect clearly apparent in the outbreak log. 29 Table 1.9: Comparison of Surveillance Reporting Rates and Outbreak Response Time EAPHLN Comparison Number of observations Counties Counties EAPHLN Comparison Intra-sub county disease surveillance 5 8 62.1% 49.0% reporting rate (weekly average, 2015) counties counties Number of days a case was first seen at a health 6 67 23.2 15.4 all outbreaks with data, facility until concrete interventions began excluding Nairobi 8 34 Proportion of outbreaks/rumours that were either Suspected cases in 5/8 26/34 confirmed or ruled out EAPHLN and comparison counties Source: DDSR 30 THE ROLE OF THE INDIVIDUAL EAPHLN PROJECT INTERVENTIONS While there was variation in the subjective priority ranking of interventions between different participants, certain trends were evident. Most participants cited SLMTA/SLIPTA (which was presented to them as a combined intervention), infrastructure/equipment, and facility improvement funds as having a relatively important impact, while supervision and operations research was consistently given relatively low priority. The eight interventions were presented to the participants in a printed form. To calculate an overall rank, as well as ranking for specific subgroups of participants (Source: Authors? ), the average ranking score for all participants was calculated 16. Appendix E contains a table that shows the individual ranks given. Table 2.1: Subjective Priority Ranking of Interventions 17 Aggregate ranks based on averaged ranking score Laboratory Laboratory Administrative Overall Technologists Managers Staff/Leaders Facility upgrading 1 3 1 1 Facility upgrading SLMTA/SLIPTA SLMTA/SLIPTA 2 2 2 3 participation participation Improvement Fund 3 1 4 2 Improvement Fund Mentorship 4 5 3 3 Mentorship Training 5 4 4 5 Training New personnel hires 6 6 6 6 New personnel hires Operations research 7 7 8 7 Operations research Supervision 8 8 7 7 Supervision Source: Authors Equipment and Infrastructure When calculating the overall rank based on the average ranking score across the fifteen interview participants, facility upgrading emerged as the overall ‘winner’. It is interesting to observe though that only three among the fifteen participants ranked this as the top most important aspect of the intervention, but because it was consistently rated high (among the top three by most respondents), it emerged on top overall, and in the aggregate ranking for the laboratory managers and administrative Staff/leaders. Participants who manage laboratories or perform administrative functions were more likely to prioritize infrastructure and equipment. Lab technologists whose responsibility is mostly the day-to-day laboratory work ranked infrastructure and equipment as priority number three overall. 16 The findings based on the ranking exercise need to be used with caution, for two reasons: (1) The sample of interview participants who participated in the exercise is small (15 individuals), and (2) calculating averages with ordinal numbers makes an assumption of equal distances between the different rank numbers, and for all participants, which can be challenged. The table presented is therefore indicative to serves as a basis for discussion. It is not a scientific measurement. 17 A green shading means that the group represented in this column ranked the intervention higher than the average, and a red shade means that they ranked it lower than the average. 31 One of the participants indicated he had selected infrastructure improvement as a priority intervention, because it serves as a basis for many other improvements. According to the respondent, good infrastructure is the minimal condition without which an efficient laboratory network is not possible. It enables Staff to provide services, and to provide them better. It increases motivation and self-confidence, allows for the acquisition of new know-how, participation in operations research, and ultimately is expected to lead to better patient outcomes. There was some equipment that was not there before the project, which is mandatory for the achievement of quality standards as defined by accreditation criteria. This included safety equipment such as a biosafety cabinets and fire extinguishers. The new laboratory building was the most visible component of the project to the outside world; it sent a clear message that ‘something has happened’, and improved the motivation of laboratory workers (see chapter on motivation below), combined with the availability of new and modern machines. The machines that were mentioned spontaneously as being a critical addition to the laboratories were the GeneXpert, the VITEK machine, and fluorescent microscopy. The laboratory information management system (LIMS) was lauded particularly in Kitale, where its potential to enable better networking, a paperless workflow, and a reduced TAT were invoked. Some weaknesses with regard to the infrastructure were also highlighted and are being addressed as part of the hand over from the national authorities to the county governments 18. One respondent noted that in at least one case, additional consultations with laboratory Staff at the time of planning for the infrastructure upgrade would have been beneficial. SLMTA Training and SLIPTA audits 19 The SLMTA/SLIPTA intervention was most frequently ranked as the top priority intervention among those supported by the EAPHLN project. Eight of the 15 participants who did the ranking put it in first place. In contrast, there were five participants who ranked it relatively low (two ranked it last), and thus the aggregate ranking position, based on ranking score averages, was number two. The difference in priority ranking may have been due to different perceptions as to whether the SLMTA and SLIPTA interventions are a starting point that will trigger other improvements (an ‘ignition key’ as one participant called it), or a follow-up intervention that makes most sense once other basic requirements such as equipment, supplies, and more basic training needs are met. Even those who ranked it as a lower priority highlighted its merits. The aggregate priority rank of this intervention was number one when looking only at interviews conducted at the EAPHLN-supported laboratories (without Nairobi). When looking only at the interviews of project and laboratory administrators, the intervention ranks in third position. The SLMTA training was offered to a larger number of laboratories in Kenya, including sites supported by the US Presidential Emergency Plan for AIDS Relief (PEPFAR), in addition to the EAPHLN-supported sites. The data do not provide clear evidence that the EAPHLN- supported laboratories scored better than others, but they did outperform their peers in the 18 Examples given by respondents that could have been optimized in some cases (county or national referral laboratories) were consistent electrical power backup including fuelling of the power generators, rooms big enough to fit the equipment, floors that can be easily cleaned, access control installed in all labs and sufficient and correct power sockets installed to cater for the high power needed for some equipment. 19 Strengthening Laboratory Management Towards Accreditation (SLMTA) is a training and mentorship program that typically lasts around a year, while the Stepwise Laboratory Quality Improvement Process Towards Accreditation (SLIPTA) is the quality improvement protocol and process that forms a basis for the SLMTA training as well as continuous monitoring and improvement after (or independent of) the training. The two were used in a complementary way in the EAPHLN project, and will be referred to jointly as SLMTA/SLIPTA intervention for the remaining part of the report. 32 SLMTA exit audit 2.5 years after the training, with laboratories with a score of less than 75 achieving 2 stars in the SLIPTA audit system, and laboratories with 75 points and more, achieving 3 stars 20. Figure 1.3: Comparison of SLIPTA Scores Between EAPHLN Supported and Other Laboratories 3 Stars 2 Stars Source: Authors The National Tuberculosis Reference Laboratory received only partial support from the EAPHLN project, and did not fare as well in the SLIPTA follow-up audits. Among the EAPHLN-supported laboratories, Machakos was the top performer, increasing the number of stars from 0 in 2011 to 4 in 2014 and sustaining the performance at the time of the evaluation. Operational Research (OR) In the view of most interview participants, operational research was not a core intervention of the EAPHLN project, possibly because there was not broad-based understanding or involvement of all personnel in this activity. In Kitale this was different; operational research appeared to have made a critical difference. The enteric microbiology research appeared to have had a substantial impact at different levels. The training (about different causes of diarrhea) was highly appreciated for both laboratory and clinical Staff, and the identification of pathogens and drug susceptibility tests revealed that clinical management of patients in the past was partially inefficient and guided by incorrect clinical assumptions. A subsequent change in clinical practice led to improved outcomes for patients with diarrhea, since they were no longer prescribed inefficient antibiotics. The joint training of laboratory and clinical Staff led to improved relationships and enhanced collaboration. While in Kitale the operational research was particularly appreciated and memorable, it had positive effects elsewhere as well. In Malindi, both a nurse and a senior hospital administrator noted that they expect a positive impact on patient outcomes. Overall, the operations research had opened a new world to laboratory Staff, who had little knowledge of and hands on experience with research before the training. The collaborative development of standardized proposals for three research areas (TB, diarrhea diseases, and malaria) and the approval of a generic proposal template in all EAPHLN member countries by institutional ethics review boards was an important element 20 Source: Table of SLMTA baseline and exit audit scores obtained through EAPHLN. The data contain SLMTA exit audit scores for 20 out of the 50 laboratories that obtained SLMTA training in Kenya. The exit audits for cohort 1 were made 30 months after the workshop, and for cohort 2 they were made 12 months after the workshop. The two data points are therefore for a different set of laboratories. For the EAPHLN laboratories (except NTRL and NML), the score was obtained in the EAPHLN peer-assessments, one 18 months after the workshop, and one 30 months after the workshop. The figure presented represents the best comparability that can be achieved with the existing data. 33 in facilitating these studies. In addition, the training provided by the KEMRI Secretariat to Staff from the EAPHLN-supported laboratories empowered them to conduct their own research to address day-to-day issues. The design and implementation of a new study approach to evaluate fluorescent microscopy, involving intervention and comparison laboratories; and the publication of a journal supplement with 11 articles were cited as important elements of good practice. Apart from these successes and positive effects, the OR program also generated important findings, such as: • The discovery of a potentially new hybrid pathogen in Northern Kenya • Understanding that there are large differences in sensitivity of the GeneXpert, between laboratory sites, probably related to differences in sputum quality due to ambient temperature and transportation and storage challenges • The realization that the correct diagnosis, treatment and follow-up of symptomatic patients who test TB-negative remains an important challenge, requiring not only the introduction of new technologies but closer collaboration between clinicians and laboratory professionals as well as closer follow up of patients. Like other aspects of the project, OR was not without challenges. At the beginning of the project, the role of the technical working group chairs and the strategy for OR was not clearly defined and some time was lost in the process of defining roles and responsibilities. The OR team reported to have had relatively limited resources, which were not sufficient to carry out regular supervision of the OR activities at the EAPHLN-supported laboratories. Thus, the enrollment of patients was slower than planned; and there was not always enough cartridges to process the GeneXpert tests. At the time of the evaluation, there was a large backlog of unexamined specimens. The success of OR at the various EAPHLN sites was partially dependent on buy-in from the hospital management, in the absence of which laboratory Staff had ascribe lower priority to it. Facility Improvement Funds The facility improvement funds, which offer financial support to the EAPHLN-supported county and referral laboratories to close operational gaps and bottlenecks identified during the SLIPTA audits, was ranked first in terms of being a priority among laboratory technologists, and ranked third overall. This innovative mechanism piloted under the project was viewed as immensely helpful, and demonstrated the importance of having access to an operating budget for laboratory Staff to do their day-to-day work, especially when it helps to bridge short-term gaps in reagent supplies. The improvement funds were also used to finance disease surveillance activities in some counties (see above). The funds were viewed as very helpful and even essential by some of the interview participants, as they provided greater autonomy to facilities. To be able to make good use of the funds, the timely channeling to facilities needs to be assured. This was a challenge, especially in the last financial year of the project. On a general note, the facility improvement funds appeared to foster autonomy and enhance accountability at project-supported facilities, highlighting the importance of providing operating budgets to laboratories. Mentorship The mentorship model used for the project in Kenya involved one week of mentorship for each site on a quarterly basis. At least one internal quality assessment was conducted before the annual peer review assessments. In addition, each laboratory benefited from one mentorship during the SLMTA training which consisted of three workshops. There was almost universal agreement among interviewees that the concept of mentorship with hands- on, on-site training, where a mentor shows how things are done, is an extremely valuable complement to classroom training. However, most interviewees also believed that the way the mentorship was structured as part of the Kenya EAPHLN project was not optimal, and therefore other project components were ranked with a higher priority. Specifically, they argued that more could have been achieved with longer and more intense on-site 34 mentorship that is organized independently of the annual SLIPTA assessments. Nonetheless, laboratory technologists interviewed expressed their appreciation and stressed that mentorship had been helpful to them as it provided hands on support without needing to travel to take advantage of training. One laboratory manager suggested that for future interventions it would be important to also establish the concept of internal mentorship, where different laboratory Staff within an institution can coach each other, depending on the training they were able to attend, which would help for the training content to cascade further. Another manager recommended that more emphasis on technical aspects of laboratory work (for example, sample production, test sampling, polymerase chain reaction - PCR) as part of the mentorship program would be useful in the next phase, as the emphasis was mostly on the quality improvement processes. Other training activities Training (other than the SLMTA program) was ranked somewhat higher by laboratory Staff than by those who work in a more administrative capacity. Different people highlighted the importance of various contents, including learning how to operate new equipment (specifically the VITEK machine and GeneXpert), safety measures, and the introduction of standard operating procedures (SOPs). Even when asked about particularly useful training they received as part of the EAPHLN program other than SLMTA, several participants highlighted themes that are related to quality improvement, such as the introduction of SOPs, “Audits”, or “documentation”. Another aspect related to training was that individuals were supported to pursue further education, including bachelors and master’s degrees. In facilities where this applied, it was perceived positively by clinical Staff, and contributed to boosting motivation and professional confidence of the laboratory Staff. In the words of one manager who echoed the views of others, “We now have nine degree holders, we are very proud”. Clinicians also remembered positively the time they benefited from EAPHLN- supported training, in particular the training that was provided as part of the operations research. There was a consensus that the joint training sessions between laboratory and clinical Staff served as a valuable platform to strengthen relationships and foster inter- professional exchanges (see also chapter on motivation). Phlebotomy training was said to have been particularly useful for medical interns in Malindi. One clinical officer suggested that the laboratory Staff provide training to them as part of their weekly continuous medical education (CME) events. Other positive outcomes included an increase in motivation (“it boosts morale, and it triggers an interest in new things”), improved work performance, and greater appreciation from clinical Staff. When asked about additional training that would be useful, the following themes appeared on the wish-list (varying between locations) of participants: • Commodity management • More technical skills, especially for molecular diagnostics • Knowledge and skills in carrying out internal quality audits • IT & LMIS skills for non-IT Staff • A larger number of Staff to be trained in SLMTA/SLIPTA Hiring of additional personnel While hiring new personnel did not emerge as a top priority when interview participants chose between the various components of the EAPHLN project, they expressed awareness that it had made an important difference. They noted that staffing shortages hindered capacity to manage test volumes and improve turnaround time, and hence “without the new personnel, things would not have worked”. In Wajir, the most remote of the EAPHLN laboratories, hiring more qualified laboratory personnel, as well as other health cadres (in the context of the newly devolved health system in Kenya) transformed their capacity to provide health services, including laboratory services. Most personnel employed by the 35 project were later absorbed into the facility as permanent Staff, a remarkable accomplishment for a project of this scope (Table 2.2). In Kitale, all six hires were integrated into the county payroll, although there were some other personnel who transferred or retired and were not replaced. In Machakos after some initial challenges it was possible to take all personnel on board. Malindi experienced the biggest challenges, as two of the EAPHLN hired laboratory technologists as well as the IT person left the site, and only four of the seven project-employed personnel moved to the county after the main project phase ended. For IT, this left an important gap, as the hospital lacked capacity both to manage the LIMS and the HMIS. It was unclear at the time of the evaluation whether EAPHLN recruited personnel at the National Microbiology Reference Laboratory (NMRL) could be absorbed. Several interview participants emphasized that the quality of the EAPHLN-hired personnel was exceptional (higher qualification, excellent skills and better motivation). Some suggested that it might be more sustainable to invest additional resources in building the capacity of existing Staff, rather than hiring new personnel, especially if it is unclear whether they can be absorbed into the government payroll at the end of a project. Project management adopted an approach that both training of existing Staff and recruitment of additional, more qualified Staff were critical to enhancing performance. 36 Table 2.2: Number of Personnel Absorbed into Regular Payroll Category Total hired Absorbed Comments Lab technologists 30 29 Scientists, 2 0 in process ICT officers, 4 2 2 left service Operations officer 1 1 Construction 1 0 monitoring Drivers 5 4 1 left service Total 41 36 Source: ? Supervision The supervision strengthening aspect of EAPHLN appeared to be less well defined or understood by the interview participants, as result of which they had less to say about it. One respondent stated that the team meetings in Nairobi at the earlier stages of the project were particularly useful in generating team work and a sense of purpose, as Staff from the different sites came together for joint budgeting and planning sessions, and exchanges of ideas and experiences. In the first phase of the project, prior to the devolution of government healthcare services, the EAPHLN coordination team conducted onsite supervision missions; later, the frequency of visits decreased. Thus, supervision was not generally seen as a priority intervention compared to other aspects of the project. The support and monitoring of the operational research activities was perceived as supervision and deemed useful, as was mentorship. One interview participant said that supervision had been helpful to them as they had been asked about their needs. Another key informant highlighted the positive effect that supervision has by underscoring the importance of the laboratory to the hospital management, since these points have more weight when they come from an outside person. An administrator pointed out that strong supervision could lead to better Staff capacity and ultimately save resources that might otherwise be required to hire additional personnel. The strengthening of supervision of lower-level laboratories was also mentioned by one participant as an important activity. 37 LABORATORY WORKERS SKILLS AND MOTIVATION Key informants were unequivocal in asserting that the interventions had positively influenced the motivation of laboratory personnel at the EAPHLN-supported laboratories, and this was attributed to several project interventions. The new automated equipment was appreciated as a professional tool, and it was also an “exciting” novelty, with the added benefit that it helped increase the laboratory workers’ prestige among clinical Staff and increased their confidence in results. Training was equally emphasized as a motivator. It enabled Staff to conduct new tests, offering a larger menu and ultimately contributing to increased service quality. Being allowed to participate in training was reported to constitute a recognition and reward; the added benefit of building new relationships was an important aspect of joint trainings – whether with clinical Staff or with Staff from other laboratories. Exposure to other laboratories and interactions with Staff from elsewhere (including teams from other countries) was explicitly appreciated by those who benefited from regional training. The team effort required to increase SLIPTA scores led to more collaboration and mutual accountability. Improving performance was a joint effort that required everyone to contribute effectively. Another motivational factor was the general improvement of the professional environment, including state of the art infrastructure, new technologies, and an enhanced supply of reagents. Increased autonomy allowed Staff to set priorities for time and budgetary allocations that led to a more effective delivery of critical services. Given access to better tools, greater autonomy, and more attention to laboratories Staff felt greater appreciation and reported being motivated to do better work. The modern and spacious work environment was said to be perceived positively by those outside the laboratory (hospital Staff, patients, and the local population), and contributed to an enhanced status of the Staff. To illustrate this, an administrator said that the number of job applications they receive at the national level had risen (“We receive thousands of applications”), which he partially attributed to the increased prestige. A laboratory manager suggested that a professional and well-equipped appearance increases the likelihood of attracting funding for research projects. The frequency and nature of routine interactions and the relationships within the laboratory and the hospital were said to play an important role for motivation. The relationship between the laboratory manager and the Staff was viewed as being an important determinant of Staff morale and performance. Regular meetings with clinicians were almost universally appreciated. Another motivation was the recognition of laboratories and laboratory workers as an important part of the broader health system. Disease surveillance officers and epidemiologists started to actively involve laboratory personnel during outbreaks. Several participants mentioned that the implicit competition on the SLIPTA checklist acted as a driver for some Staff, such as at the Machakos laboratory. Respondents suggested that a well-functioning PBF system with bi-annual assessments could have a further positive effect in terms of stimulating healthy competition and providing incentives to focus on results. Since PBF payments under the project scheme had not yet been paid out at the time of the evaluation, these statements were based on respondent views and on exchanges with colleagues in Rwanda. Interview participants from the laboratories also described some barriers or bottlenecks related to sustaining motivation. In Wajir, the fact that many of the Staff came from other parts of the country had an adverse impact on morale and team work. Perceived and real insecurity weighed heavily on those working in fragile and difficult environments at several 38 project sites. The promise of gaining experience and knowledge, especially in the case of the additional workload stemming from the operations research, was said not to be sufficient in the long term to maintain motivation. For the quality audits (SLIPTA assessments), regularity and consistency is very important to sustain motivation. In Malindi, there was a visible effect on compliance with quality standards when the assessment was skipped in one year. A combined analysis of the accounts given in the interviews leads to the emergence of three key motivational factors from the EAPHLN interventions: (1) working in a conducive environment with the right set of professional tools; (2) benefitting from opportunities for individual growth and intellectual stimulation; and (3) interacting with clinicians and hospital directors more regularly which can generate positive spillover effects in terms of improved and expanded relationships (Figure 4). Figure 1.4: Key Motivational Drivers Identified in Kenya Source: ? Various EAPHLN project interventions supported these three factors (professional environment, individual growth, and relationships) often more than one at a time, resulting in mutually reinforcing effects. While on the surface many of the interventions may have to target predominantly the professional environment – especially the facility upgrading, additional human resources and facility improvement funds – the same interventions also had a positive impact on personal growth and relationships. Trainings and exchanges with professionals outside the laboratories contributed to all three factors: a conducive professional environment, individual growth, and strengthened relationships. New skills Different skill sets acquired as a result of the EAPHLN training were highlighted at the different sites. Laboratory technologists emphasized the importance of increased Staff capacity; for some of them this was more of a priority than the acquisition of new machines. As noted in previous sections, the importance of documentation was emphasized by interviewees in Machakos; in Kitale, the establishment of SOPs led to a change of mindsets, while learning related to operate the new equipment was a highlight in Malindi; and the phlebotomy training received several positive mentions. 39 The training on the stepwise approach to laboratory accreditation (SLMTA/SLIPTA) was appreciated by laboratory managers everywhere; it was said to have had a transformational effect on Staff, leading to an emphasis on quality that was previously rare. The SLIPTA Staff meetings, as well as the interactions with clinicians led to a new feeling of ‘joint purpose’. 40 PROFESSIONAL RELATIONSHIPS AND LABORATORY PERFORMANCE Laboratory workers as well as clinical Staff explained that the improved laboratory services, in combination with the increased visibility of the facility had a positive effect on relationships with clinical Staff. One interview participant noted that “they have started to talk to each other, which is extremely important.” As a result of the project interventions, clinicians and laboratory technologists were more likely to discuss medical cases and there was greater use of laboratory testing for diagnosis and treatment of patients, paving the way for evidence-based clinical decisions. While more exchanges and improved relationships between laboratory Staff and clinicians were observed, they were described as somewhat fragile and dependent on the laboratories’ ability to continuously deliver satisfactory services. In places with a high turnover of clinical Staff (such as Malindi and Wajir, where a large proportion of medical interns change every year), laboratory workers were said to still not be viewed with sufficient respect by some of the newly arriving clinicians. Broadly speaking, the respect of clinical colleagues for laboratory Staff was at risk when performance declined, especially for turnaround times and test availability. Based on the combined accounts of both laboratory and clinical Staff, three main factors can be recognized that led to a strengthened relationship between laboratory professionals and clinical Staff at the EAPHLN-supported laboratories: (1) New tests and increased trustworthiness of the laboratory The acquisition of modern equipment, implementation of stricter rules, additional formal training and acquisition of academic titles, and introduction of higher performing automated technology had the effect that clinical Staff perceived the laboratory workers as being more productive. The clinician/laboratorian relationship was viewed as being dependent on the efficient delivery of high quality, accurate laboratory services. One clinical officer asserted that the laboratories had become useful “because their results are consistent with our empirical diagnosis”. Other clinical workers noted that the laboratory results had helped them to better manage their patients because the laboratory findings showed that they had to change the therapy. (2) Joint activities between laboratory and clinical Staff Several joint activities were organized as part of the project, mostly in the context of training and communications. This included meetings to inform hospital Staff about the new quality standards (for example, the new registration process, order form requirements, access restrictions), meetings in the context of CME and collaboration on operations research. These interactions contributed to enhanced appreciation of laboratory personnel and improvements in the relationship between clinicians and laboratory Staff. One laboratory technologist in Kitale asserted that “they almost see us at par now, as we also assist patients”, and in Wajir, for the first time laboratory workers and clinicians sat together to discuss how they could jointly improve outcomes. 41 (3) Increased routine interactions The broader test menu and increased demand for testing based on automated technologies led to more interactions when samples were delivered or picked up in the laboratory. This increase in face-time in itself had a positive impact on the relationship. Furthermore, as a result of the new quality management system, new channels of communication were established, and they were used more regularly. This included an improved ordering and complaint process, telephone lines, and frequent circulars with updates from the laboratory. In conclusion, findings suggested that joint trainings and inclusive decision making (where clinicians are informed and involved in project activities), as well as the set-up of systems that increased routine face-to-face interactions between laboratory and clinical Staff have a beneficial effect on relationships, and can contribute to improved clinical decisions. Enhanced recognition has a motivational effect on laboratory Staff, and may prompt them to be more responsive to the needs of clinical Staff. 42 STAKEHOLDER CONCERNS ON SUSTAINABILITY The discussion of threats to sustainability aimed to help stakeholders identify priorities and mitigation measures. There were substantial concerns among interview participants that some of the progress achieved might not be fully sustained in the long term. There was also a perception voiced by a few respondents that there were some issues that felt “unfinished” or completed but not yet embedded in practice. Perhaps the strongest expression of this sentiment was articulated by one participant who stated that “the new infrastructure feels just like a ‘building’ – what we want now is a fully functional lab”. Another participant expressed uncertainty about whether the foundation laid by the project was strong enough to survive in the absence of continued project support. Key bottlenecks that were seen as a threat to continuity were related to: (1) lack of a regular supply of reagents, which impedes testing; (2) lack of service contracts for all equipment which can lead to disruptions in service provision; (3) delays in the delivery and installation of required equipment and; (4) concerns over maintaining adequate staffing levels. Respondents noted that the expected reduction in resources towards project closure might have negative effects on: • Quality improvement process: The positive trend observed in the SLIPTA audits could stall or reverse in the absence of regular audits, continued training and facility improvements. • Outbreak preparedness and response: If access to funds that support quick outbreak response is limited, due either to slow processes or an absence of financial resources, outbreak preparedness could suffer a ‘serious blow’ (in the words of a participant). • TAT and client satisfaction: These indicators of performance are affected by supply bottlenecks, equipment breakdowns and Staff shortage. Reagent supply Delays in the supply of reagents was the concern emphasized most heavily by interview participants as affecting performance and sustainability. A typical explanation given was that after the devolution of the health system and a reduced flow of laboratory improvement funds from the EAPHLN project, the laboratories started experiencing stock-outs, and reported long delays in the procurement process. This started to have an adverse effect on client satisfaction, the relationship with clinical Staff, and impeded the enrollment of patients in the operations research studies. In cases where no supplies were available to carry out testing, patients had to be sent to other facilities (typically in the private sector). Some of the tests that were reportedly affected by stock-outs were hematology, biochemistry, and molecular biology. Respondents at one of the sites reported that they ran into that problem despite having a number of equipment on placement contracts with manufacturers, since the contractually agreed payments were late. 43 Equipment As described earlier, the new automated equipment played a critical role in expanding the range of tests offered, increased clinician trust in results, and positively influenced Staff motivation. However, some of the diagnostic and video communication equipment that was procured under the project had not yet arrived on site at the time of the evaluation, or the installation and commissioning was delayed. Gaps in terms of having a fully equipped lab were also noted by respondents, including lack of power backup, disruptions in air conditioning, absence of back-up machines in case of breakdowns and limited or no integration of the laboratory information management system (LIMS) with hospital systems. According to the accounts given, no service contracts (beyond the initial warranty) were in place for equipment maintenance and repair, which could compromise sustainability. Based on an update provided more recently, several of the identified challenges have been resolved or are being addressed. All video equipment had been delivered to the sites, and procurement for the required internet bandwidth was in process. The construction works were completed in Busia, which paved the way for the equipment to be installed, which was scheduled for January 2017. Training for LIMS super-users had been included in the project’s annual plan, and software development for improved integration and interoperability was budgeted for the financial year 2016/17. Backup machines for haematology and biochemistry were in the procurement plan of the ongoing year, as well as histology machines and cancer screening equipment for four sites. Human resources As described earlier, the absorption of most project-hired laboratory personnel into county Staff payroll augurs well for sustainability. Laboratory Staff asserted that the broader menu of tests offered and the increase in test volumes combined with improved efforts to boost quality increased workloads. The greater workloads were particularly evident during night- shifts and over weekends, and led to periodic delays in the delivery of test results. A manager explained that he would like to spend more time on management functions, including individual Staff reviews and support, but needs to spend much of his time doing hands-on work at the bench. Handling the reception of specimens was highlighted as problematic at two sites, as the function of a receptionist had to be taken by qualified laboratory Staff. Two specific skills and qualifications were said to be in need: qualifications for PCR testing, and IT. According to one of the project administrators, concerns related to Staffing shortages were mostly in anticipation of future workloads, while at the current time, Staffing levels were adequate. 44 RECOMMENDATIONS Participants offered suggestions on how to strengthen performance and ensure sustainability of project interventions. Recommendations are grouped into three broad categories focusing on strategies for mobilizing predictable financing for laboratories; deepening and broadening the results oriented culture; and strengthening the clinician/laboratorian interface which is critical to success. Recommendation 1: Provide information to county health leaders about the benefits of strong laboratory services, and urge county authorities to allocate and earmark budgetary resources to consolidate and sustain investments. Potential strategies for laboratory financing • Establish a county budget line to cover recurrent costs of laboratories, including a stable supply of reagents (especially for low-volume tests that do not meet the criteria for equipment placement) and equipment service contracts. • Enter into agreements to pool funds with neighboring counties, whose citizens benefit from the hub laboratory services, if an efficient system for inter-county specimen transport is put in place. • Advocate for continued financial support from the national government in the form of equipment maintenance and repair, and Staff secondment, especially for regional laboratories that are conceived as spokes of a national referral system. • Support an enhanced coordination of donor support provided by different international partners for laboratory strengthening at a regional level (county clusters) to foster optimization of resources and reap synergies and cost efficiencies. • Consider county-level financing mechanisms that allow facilities to retain a part of the funds generated through cost recovery to be re-invested at the laboratory operations • Evaluate contracting arrangements that may reduce the financial burden on county governments by purchasing certain services rather than establishing in house capacity, particularly for low volume, highly specialized tests. One way of doing this is equipment placement, where a laboratory device manufacturer places their diagnostic machines in a laboratory at no upfront cost for the equipment, and provides maintenance and repair services, in exchange for a contract that obliges the county to buy a specified volume of reagents over a time period 21. 21 For placement, the price of the reagents covers equipment and servicing costs. Such arrangements typically remove the need for high capital investment/upfront costs and minimize equipment breakdown time. 45 Recommendation 2: Broaden and deepen the results-driven environment established during the first phase with more effective use of data, stronger accountability mechanisms, where the voice of the client plays an increased role, and more systematic use of performance based financing. This requires additional investments to strengthen and connect IT systems and bolster Staff capacity to use them. The regular, peer-driven SLIPTA quality audits, and the training in management processes instilled a culture focused on quality results, established a pathway for laboratories to progress towards accreditation, and contributed to Staff motivation. This results-oriented and transparent process can be further expanded to include a more rigorous effort to monitor and track client satisfaction and to incorporate additional performance indicators. Potential activities to enable a results-driven environment • Continue, strengthen and expand SLIPTA assessments, using a mechanism of cross-county verification. • Regularly and systematically administrate the standard client satisfaction questionnaire that was developed by ECSA-HC in 2016. The questionnaire should be accompanied by guidelines on the frequency, sampling methods, data collection, data entry and sharing of results. • Devise a set of key performance indicators (KPI) based on routine data, that include turnaround time for a rotating subset of tests; test volumes; stock-outs; and equipment downtime. These indicators would be reviewed on a quarterly basis and comparisons and trends shared with the complete teams of participating laboratories • Intensify technical support and training for laboratories with a new or recently introduced LIMS. A comprehensive change to the electronic system should be encouraged and incentivized. This includes the correct and immediate registration of all specimens at the time of their first receipt by the laboratory, and the setup of reports for the extraction of statistics. Validation checks at the time of data entry should be introduced. An integration of laboratory IT systems with the hospital information systems would increase its usefulness and impact. Interoperability of the laboratory information management system with the existing DHIS2 software and a direct link to the Ministry of Health Disease Surveillance and Response Unit should also be considered. • Consider an increase in resource allocation for PBF for laboratories, to ensure adequate incentive amounts, to be used for laboratory improvements and Staff bonuses. Ensure regularity in conducting assessments and incentive payments, and consider broadening the set of indicators used for the PBF mechanism. Performance reviews could address multiple dimensions of performance, including SLIPTA scores, turnaround times, test volumes, and client satisfaction. 46 Recommendation 3: Continue to invest in joint learning initiatives that bring together clinicians, surveillance officers, researches and laboratory Staff, and organize exchanges between different laboratory sites. The evaluation showed that joint learning has a positive effect on motivation, as it fosters a professional environment, promotes individual growth and strengthens professional relationships. Laboratory Staff at the EAPHLN-supported facilities can possibly be engaged as mentors and assessors for other laboratories. Using these Staff to cascade knowledge and skills would expand their own abilities and increase the uptake of good practices. An on- site mentorship model, where experts coach laboratory managers and technologists, may have a more discernible effect on Staff motivation, knowledge, and performance. A model where specialists are engaged to spend 2-3 weeks in a laboratory every month for a duration of several months could be introduced as part of the national support to counties. 47 APPENDIX A: EVALUATION METHODOLOGY After deliberations with the project team and a review of prior similar work, consensus was reached to assess volumes, timeliness and accuracy based on a subset of laboratory tests that represented different laboratory departments and varying extents of routine or specialization. The design intended to include tests that were covered by new equipment as well as tests that were not specifically targeted by the project. The final range of tests included in the analysis was determined by data availability and completeness. The design aims at assessing test timeliness through turnaround-time obtained from laboratory records, test accuracy through nationally consolidated records of external quality assessments / proficiency testing (PT), and client experience through existing data from satisfaction surveys of clients, both patients and clinical hospital Staff. SAMPLE AND DATA COLLECTION Performance-related questions The performance assessment was designed to provide insights about the relationship between the intervention and laboratory performance. If possible, facility characteristics that may confound the results should be controlled for. Outcome variables, variable of interest (intervention), as well as potential confounding variables are summarized in Table 2.3. The limited data availability and completeness allows for a regression analysis only for two dimensions: EQA and surveillance performance. For all other outcome variables, the sample sizes were too small and allowed for a descriptive analysis only. Table 2.3: Control Variables for the Performance Assessment Variables Variable type Operationalization 1 EAPHLNP Interventions Variable of interest Yes/no dichotomous variable Confounding 2 Catchment population size County population variable Confounding 3 Epidemiologic characteristics HIV, Malaria, TB prevalence variable Confounding 4 Distance from Nairobi Driving distance in km variable Confounding 5 Hospital size Number of beds variable Adherence to quality SLIPTA score 6 Outcome variable (only available for EAPHLN facilities) standards Trend in test volumes (growth) for a range of 7 Testing volumes Outcome variable representative tests Trend in mean monthly TAT for a range of 8 Timeliness Outcome variable tests, as per data availability Per-laboratory description of survey findings, 9 Client experience Outcome variable as reported by laboratory managers Routine EQA tests performed under the 1 Test accuracy Outcome variable HuQAS PT program, as per the laboratory 0 participation 1 Surveillance performance Outcome variable Facility reporting rate 1 1 Outbreak preparedness / Outcome variable Response time (number of days) 2 response Source: ? Data sources and sample 48 The sources and quality of the data obtained are described in detail Appendix B below. Secondary data from all medical laboratories of public health facilities in the country that provide services at the same level of care as the EAPHLN laboratories (former Level 4 hospitals) was included for the EQA performance assessment. For test volumes, a purposive sample of comparable laboratories was selected. Qualitative method The qualitative component of the evaluation was carried out in the form of 25 in-depth stakeholder and expert interviews. Conversational-style interviews were conducted based on written discussion guides. The interviewer, who was external to the project, probed beyond the written questions where necessary, to learn about specific details, tangents, and unexpected insights. All interviews were audio-recorded with a digital device, and transcribed verbatim. Participants were informed that their responses will remain confidential, and that their participation is voluntary. Table 2.4 shows the key informant interviews carried out for the qualitative assessment. They were conducted on site in three laboratories (Malindi, Kitale and Machakos), in Nairobi, and during an EAPHLN planning meeting in Nakuru. The interviews provided a range of different professional (subjective) accounts of the project experience, and how it influenced the work, skills, motivation and relationships in the respective facilities. The aggregation of a range of different key informant perspectives from different contexts mitigates, to some extent, the risk of respondent bias, but it is inherent to the nature of the qualitative methodology that it does not provide fully objective, verified evidence. Table 2.4: Final Qualitative Sample Achieved in Kenya Respondent Category Number EAPHLN laboratory manager 4 EAPHLN laboratory technologist 3 Clinical Staff who order lab tests One nurse, one doctor, four clinical 6 officers Hospital Director 1 County health management team Two county directors of health, two county 5 laboratory technologists, one county disease surveillance officer NRL Manager (Nairobi) 1 EAPHLN laboratory consultants/mentors 2 Head of the project and project officer 2 Technical working group chairperson 1 TOTAL QUALITATIVE SAMPLE 25 Data Analysis The analysis of the quantitative data was done with the support of a professional statistician. With the exception of the EQA and the surveillance response rate data, the sample sizes were too small to conduct tests of statistical significance. For the qualitative part of the evaluation a content analysis of interview transcripts was carried out. The transcripts were coded and re-organized according to the research questions. The report does not personally identify any of the participants, to protect confidentiality. 49 APPENDIX B: DATA SOURCES AND DESCRIPTION Turnaround time Data source: Laboratory Information Management System (provided by National Public Health Laboratories, NPHL). Sites included: Busia, Kitale, Machakos, Malindi (intervention), Nakuru County Hospital (comparison). Data were only available from a single comparison site, and therefore it was not possible to match hospital characteristics; however, to address the problem of initial differences, the growth rate of TAT (average percentage change between consecutive months, month t+1 – month t) was analysed rather than absolute times. Data description: The original dataset contained average monthly turnaround times (in minutes) for 183 different tests in the period from May 2015 - June 2016. The cleaned dataset used for the analysis included 11 tests that were reported in at least one of the intervention sites and had 30 or more observations in total across the different laboratories. The data from Kitale and Nakuru cover the entire observation period, while the other facilities only have data for a part of the period; Busia only had data for 2 months. The total number of observations in the analysis dataset (average monthly turnaround time including outliers) was 475; 125 are from the comparison site. The use of the LIMS is still in its infancy in the project sites, and the turnaround-time data recorded in the system are of uncertain validity. One challenge reported from the sites is that samples are sometimes logged at the time when processing begins, instead of the time when the specimen is received. Comprehensive and complete data were available from one comparison site, while there were a lot of missing data in the intervention sites. The quality of the quantitative evidence on turnaround time is poor. The data contained outliers, but it was not possible to determine an ideal cut-off rate. Therefore, the top 10 percent and the bottom 10 percent of observed TAT increase/reduction rate were removed for the final analysis to reduce the influence of outliers. Test volumes Data source: National District Health Information System (DHIS2) Sites included: Kitale, Busia, Wajir (intervention sites); Homa Bay, Kericho, Kisii Hospital, Kisumu County Hospital, Jaramogi Oginga Odinga Hospital, Mandera, Olkalou, Nyeri, Samburu, Hola, Siaya, Alupe, Garissa, Marsabit (comparison sites). Comparison sites were selected purposively to represent a comparable setting; a larger number was selected to boost the number of observations and create a stronger data basis. Data description: The dataset contained quarterly test volumes as reported in the routine health management information system for 13 tests that were selected by consensus, taking into account data availability. The original dataset (including outliers and tests with low observation numbers) had 1,404 observations of quarterly test volumes. There was a substantial proportion of fields with missing data, especially in remote facilities and including the EAPHLN sites. DHIS 2 does not contain data on test volumes for Machakos and Malindi. The data basis for this analysis is weak due to the large percentage of missing data and limited number of observations for each individual test. For the analysis, the top 10 percent and the bottom 10 percent of observed quarterly test growth rate were removed for the analysis to reduce the influence of outliers. 50 Test accuracy Data source: Human Quality Assessment Services (HuQAS) test database (received through NPHL). The East Africa Regional Quality Assessment Scheme (EA-REQAS) and the HuQAS are two of the most used laboratory Proficiency Testing (PT) providers in Kenya. All the EAPHLN project sites have been participating in the two proficiency testing schemes. The EA-REQAS is provided by the African Medical Research Foundation (AMREF) while HUQAS is a Kenyan registered not-for-profit organization and has been offering (PT) services since the year 2000. PT panels include serological, Haemoglobin estimation, Chemistry, haematology and CD4 tests among others. The role of Proficiency testing is to give objective information and advice to laboratories on the quality of their analytical performance as a base for improvement on the accuracy and reliability of their results. The PT multiple samples are sent to laboratories three times a year for analysis and/or identification. They are analyzed at the laboratory and results returned to the provider through an online portal. Each laboratory’s results are compared with those of other laboratories in the group and/or with an assigned value, and reported to the participating laboratories. The performance reports are posted to individual labs online. Corrective actions are undertaken after discussions of the results with the laboratory management and results used to direct improvement efforts. Proficiency testing helps timely detection of equipment failures, identification of faulty reagents, reviewing of Staff training and competencies. This ensures maintenance and improvement of inter-laboratory agreement therefore raising standards as well as instilling confidence in lab test results. Sites included: Busia, Kitale, Machakos, Malindi (intervention sites); 35 district hospital laboratories and 1 health centre (comparison sites). Data description: The HuQAS data set network statistics summarizes the test results from 5 tests events for all laboratories covered by the assessment (April and July 2014; April, July and November 2015). The PT testing is made across a broad number of tests in various laboratories, but not for all of them did the EAPHLN intervention sites participate. Tests types assessed in the intervention sites included Chemistry/Immunoassay, Haematology, Malaria, Mycobacterium Ziehl Nelsen stain, Blood parasites, and Lymphocyte Immunophenotyping. For each sample tested by the hospitals, the reported result is classified as acceptable or unacceptable. Another classification was “not evaluated”, which was typically the case when testing was suspended due to reagent stock out or equipment breakdown. Among the specific tests where EAPHLN sites participated, there was a total of 10,624 records across all sites (9,661 in comparison sites and 963 in EAPHLN sites). 51 Disease surveillance and outbreak response Data source: Ministry of Health - Disease surveillance and response unit (DDSR) 1. Outbreak and rumour log, 2015 - 2016 2. 2015 County performance in intra sub county reporting rate (from surveillance database) Sites included: The comparison was made at the county level (counties with an EAPHLN supported laboratory vs. comparison counties). A purposive sample of comparable counties were included in the sub county reporting rate performance assessment: West Pokot, Kwale, Kitui, Makueni, Mandera and Marsabit. For the outbreak response time, all counties that had reported an outbreak or rumour in 2015 or 2016 were included except Nairobi where the largest number of outbreaks was reported. Table 2.5 shows the number of incidents reported in each county – highlighted in grey are the counties with an EAPHLN supported laboratory. For the comparison of proportions of confirmed cases, only similar counties were included (Bungoma, Kitui, Makueni, Bungoma, Mandera, Marsabit, Turkana, Uasin Gishu and West Pokot); the proportion of confirmed cases in these counties was identical (76 percent) to the one in all non-EAPHLN supported counties (without Nairobi). Table 2.5: Number of Outbreak/Rumor Incidents Reported by County, 2015 - June 2016 County Reports County Reports County Reports County Reports Baringo 2 Kiambu 4 Migori 4 Nyeri 1 Bomet 1 Kisii 1 Mombasa 3 Siaya 3 Bungoma 2 Kisumu 2 Murang'a 2 Taita Taveta 1 29 Embu 2 Kitui 4 Nairobi (excluded) Tharaka-Niithi 1 Garissa 11 Lamu 1 Nakuru 6 Trans Nzoia 1 Homa Bay 3 Machakos 3 Nandi 1 Turkana 3 Isiolo 3 Makueni 3 Narok 1 Uasin Gishu 3 Kajiado 2 Mandera 12 Nyamira 2 Wajir 4 Kakamega 1 Marsabit 2 Nyandarua 1 West Pokot 5 Source: DDSR Method for data processing and analysis: For the reporting rate, the 53-week average was calculated and compared between counties with an EAPHLN supported laboratory and those without. For the outbreak response, the duration of days between the DDSR reported “Date concrete interventions began” and the “Date a case was first seen at a health facility” was calculated. Negative values were considered erroneous, and ignored for the analysis. The number of observations as well as the face validity of the data for outbreak response does not allow to consider it as solid evidence. 52 APPENDIX C: DETAILS ON AVERAGE TAT RETRIEVED FROM LIMS Table 2.6 shows the average turnaround times that were provided by the LIMS report. For each test, the first line shows the average turnaround time reported in hours; the second line shows the standard deviation in minutes between monthly averages. This show how much the values differed between months – the higher the number, the less likely that the reported values are reliable. The third line indicated the number of months in which an average TAT was available for each test/laboratory. For Busia, very few observations were available, while Kitale and Nakuru had the most comprehensive/complete data. Table 2.6: Average Turnaround Times Reported in LIMS and Standard Deviations Laboratory Test Kitale Machakos Busia Malindi Nakuru 0.9 1.7 5.0 236.3 Blood Slide for Malaria 17 108 176 21,077 14 10 2 10 1.4 9.4 82.8 3.6 Erythrocyte Sediment. 24 758 7,407 95 Rate 13 10 10 14 2.4 0.6 313.8 4.7 Fasting blood sugar 148 34,930 335 8 1 10 13 0.9 37.4 2.6 299.3 8.4 H Pylori 24 4,071 114 24,646 501 14 8 2 6 14 0.5 4.3 7.7 242.1 2.1 Hb 16 253 401 22,183 53 13 9 2 10 14 1.0 4.1 2.2 319.2 1.5 Pregnancy Diagnostic 29 140 12 29,465 24 Test 14 10 2 10 14 5.2 6.4 2.3 341.9 6.1 Random blood sugar 462 579 78 35,374 315 14 10 2 10 14 0.9 5.4 170.9 2.7 Rheumatoid Factor 6 325 10,221 60 12 10 10 14 6.3 394.2 27.1 Sickle Cell Test 468 31,981 1,243 7 10 14 1.3 2.0 2.0 232.7 Stool Ova and Cysts 19 110 20,077 14 5 1 10 1.0 6.3 2.3 2.8 Urinalysis 13 225 61 92 14 11 3 14 43 APPENDIX D: ASSESSMENT OF EQA PERFORMANCE (ALTERNATIVE ASSESSMENT METHOD) For the assessment of EQA performance in this report, any tests that were not submitted to the EQA agency (marked as “NE” - not evaluated - in the HUQAS summary sheet) were excluded from the analysis. An alternative approach is to include these tests and classify them as a “not acceptable” – with the logic that the “NE” result suggests that these tests were not being performed in the laboratory at the time of the assessments (for example due to equipment down-time or lack of reagents), or that the laboratory failed for other reasons to participate in the test. Non-availability of a test that is on the service menu could be interpreted as a disadvantage to a client or clinicians, and a sign of suboptimal quality of laboratory services. The table below shows that “NE” is categorized as an unacceptable test result. Table 2.7: Alternative Analysis of EQA Results 44 APPENDIX E: INDIVIDUAL RANKING GIVEN TO THE DIFFERENT PROJECT INTERVENTIONS 22 1The participant function “Administration/management” was used to group together the data from the project coordinator, the head of NPHL, and two laboratory specialists with administrative functions at the county level. Fields with grey shadings show number 1 priority ranks. 22 The participant function “Administration/management” was used to group together the data from the project coordinator, the head of NPHL, and two laboratory specialists with administrative functions at the county level. Fields with grey shadings show number 1 priority ranks. 45 This paper presents findings from a performance evaluation of laboratories in Kenya supported under the East Africa Public Health Laboratory Networking Project (EAPHLNP). The aim of the evaluation was to document progress and lessons learned, to enhance performance and foster sustainability. This paper is designed to stimulate an internal discussion for stakeholders involved in the provision of public medical laboratory services in Kenya at the county and national levels, with a particular focus on those involved with the EAPHLNP. The evaluation reviewed five different aspects of performance: test turnaround time, volume, accuracy, client satisfaction and scores achieved in peer-audits on the stepwise laboratory improvement process towards accreditation (SLIPTA). The evaluation also examined the perceived importance of the various project interventions, laboratory worker motivation, and the relationship of laboratory workers with their clinical counterparts. The assessment used quantitative information from routine service statistics, and qualitative data from key informant interviews that captured stakeholder perceptions, concerns, and insights to help understand and contextualize findings. The evaluation found an increase over time in the SLIPTA scores at the EAPHLN-supported laboratories, along with indications that client satisfaction and test accuracy improved. Most personnel employed by the project were absorbed as permanent Staff. There were no conclusive findings about laboratory performance in terms of turnaround time, test volumes, and effect on outbreak response. Three factors were identified that led to strengthened relationships between laboratory professionals and clinical Staff: (i) availability of new tests, greater confidence in laboratory results, and enhanced respect for laboratory workers; (ii) joint training activities between laboratory and clinical Staff; and (iii) an increased number of routine interactions. Given the rapidly devolved context in Kenya, the authors recommend that stakeholders share key findings with county governments to inform the design of strategies for continued funding of high-quality public health laboratory services; discuss and expand use of results-driven approaches, including promotion of more effective use of data and of accountability mechanisms; and expand joint learning initiatives that bring together clinicians, surveillance officers, researches and laboratory Staff. ABOUT THIS SERIES: This series is produced by the Health, Nutrition, and Population Global Practice of the World Bank. The papers in this series aim to provide a vehicle for publishing preliminary results on HNP topics to encourage discussion and debate. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author(s) and should not be attributed in any manner to the World Bank, to its affiliated organizations or to members of its Board of Executive Directors or the countries they represent. Citation and the use of material presented in this series should take into account this provisional character. For free copies of papers in this series please contact the individual author/s whose name appears on the paper. Enquiries about the series and submissions should be made directly to the Editor Martin Lutalo (mlutalo@ worldbank.org) or HNP Advisory Service (healthpop@worldbank.org, tel 202 473-2256). For more information, see also www.worldbank.org/hnppublications. 1818 H Street, NW Washington, DC USA 20433 Telephone: 202 473 1000 Facsimile: 202 477 6391 Internet: www.worldbank.org E-mail: feedback@worldbank.org