External quality assessment of laboratory performance in bacteriology in the Eastern Mediterranean Region, 2011–2019

PDF version

Raynal C. Squires,¹ Ali Al Abri,² Azza Al-Rashdi,2 Aisha Al Jaaidi,² Omaima Al Harthi,² Sebastien Cognat,³ Christopher Oxenford,³ Humayun Asghar,⁴ Karen Nahapetyan,⁵ Amany Ghoniem,¹ Frank Konings³ and Amina Al Jardani²

¹Public Health Laboratories, World Health Organization Regional Office for the Eastern Mediterranean, Cairo, Egypt (Correspondence to: R.C. Squires: This email address is being protected from spambots. You need JavaScript enabled to view it.). ²Central Public Health Laboratory, Ministry of Health, Muscat, Oman. ³Public Health Laboratory Strengthening, WHO Headquarters, Lyon, France. ⁴Polio Eradication, World Health Organization Regional Office for the Eastern Mediterranean, Cairo, Egypt. ⁵Infectious Hazard Management, World Health Organization Office for Europe, Copenhagen, Denmark.

Abstract

Background: Since 2007, national public health laboratories in the WHO Eastern Mediterranean Region (EMR) have participated in a regional external quality assessment scheme in bacteriology to improve testing proficiency.

Aims: To assess laboratory performance in bacteriology in the EMR between 2011 and 2019 using the regional external quality assessment scheme.

Methods: We analysed the accuracy of participant-reported data in bacterial identification, Gram stain microscopy, and antimicrobial susceptibility testing. For each category, we assessed the performance over time, the performance on multiple organisms, and whether a laboratory repeatedly failed to attain satisfactory results.

Results: Between 2011 and 2019, 70% of laboratories achieved satisfactory performance for bacterial identification and antimicrobial susceptibility testing, and 85% performed satisfactory Gram stain microscopy. Testing did not improve on multiple organisms and results were consistently low for some pathogens and test categories. Twenty-nine percent of laboratories underperformed throughout the study period.

Conclusion: The unchanged performance over time and underperformance of laboratories highlight the need for improvements in the regional external quality assessment scheme. Participating laboratories and WHO need to work more actively to strengthen the problem areas.

Keywords: bacteriology, Eastern Mediterranean Region, external quality assessment scheme, laboratory performance, public health laboratories

Citation: Squires RC; Al Abri A; Al-Rashdi A; Al Jaaidi A; Al Harthi O; Cognat S; et al. External quality assessment of laboratory performance in bacteriology in the Eastern Mediterranean Region, 2011–2019. East Mediterr Health J. 2022;28(12):856−862. https://doi.org/10.26719/emhj.22.096
Received: 08/03/22; accepted: 11/08/22

Copyright © Authors 2022; Licensee: World Health Organization. EMHJ is an open access journal. This paper is available under the Creative Commons Attribution Non-Commercial ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo).

Introduction

Quality management of all processes and procedures helps laboratories achieve the best results possible. Assessment is a key component of quality management, allowing laboratories to verify proficiency in specific areas and detect sources of error (1). To verify testing proficiency, laboratories conduct internal checks and participate in external quality assessment schemes (EQASs) to compare their test results with a source outside the laboratory. Cycles of assessment and correction should eventually lead to improved performance over time. WHO has organized several EQASs regionally and globally to help laboratories assess and improve their testing quality, and to gauge the capacity of Member States in various disciplines and for detecting various pathogens (2–5).

The WHO Eastern Mediterranean Region (EMR) comprises 21 countries and the Occupied Palestinian Territories (6). In 2004, directors of national public health laboratories in the EMR requested that WHO implement an EQAS for microbiology. WHO began working with the Central Public Health Laboratory (CPHL) of Oman and Reference Health Laboratory of the Islamic Republic of Iran to provide a regional EQAS for these laboratories, with modules in bacteriology, serology, mycology, and parasitology. The CPHL of Oman produced the bacteriology module and the Reference Health Laboratory of the Islamic Republic of Iran produced the other modules. WHO piloted the scheme in 2005 and began its first official round in 2007, with 1 to 2 rounds occurring yearly since then. Since the inception of the scheme, WHO has supported the providers with technical input, procurement of diagnostic reagents and materials, and cost of shipping the modules.

EMR Member States are diverse in income, political stability, and health system development, and possess laboratory systems ranging from basic to advanced. Several Member States face infrastructure and resource losses resulting from protracted emergencies and embargoes that complicate procurement and movement of equipment, specimens, and reagents. These difficulties have adversely affected their participation in the regional EQAS, with some laboratories not joining for many years.

The aim of the current study was to assess the performance of laboratories participating in the regional EQAS. Such insight is needed if the scheme is to remain a useful resource for the EMR. Our study covered performance in the bacteriology module for each round of the scheme between 2011 and 2019, after the content of the panels (groups of samples in a round of testing) became consistent.

Methods

Preparation for the regional EQAS

For the bacteriology module, the CPHL of Oman typically dispatched 7 or 8 simulated clinical specimens and pure cultures linked to clinical case scenarios twice yearly (May and October). These contained pathogens of public health concern in several bacteriological disciplines, including Gram stain microscopy, antimicrobial susceptibility testing (AST), enteric pathogens, and meningeal pathogens. The CPHL reserved about 5% of each panel for internal quality control, which involved full identification and characterization of specimens expected from participants, and daily confirmation of pathogen viability in transport medium during the 4-week open period of the scheme. For external quality control, the laboratory shipped panels in parallel to 2 International Organization of Standardization 15189-accredited reference laboratories, which identified specimens and commented on the quality of the preparation.

Data collection and feedback to laboratories

For each specimen, participating laboratories reported to the CPHL of Oman the culture media and identification methods used and the results for microscopy, serotyping, and AST, as relevant. The CPHL scored each item for accuracy and completeness using a 4-point scale established for clinical microbiology EQA (7), and took into account different methods used. The CPHL graded, but did not score, late results and excluded poorly viable specimens from scoring for all participants. The CPHL distributed individual and global reports 6–8 weeks after the regional EQAS ended. Each laboratory had a coded identifier for confidentiality.

Data analysis

We received and evaluated aggregate laboratory scores for each of the 3 broad categories of tests: identification (by culture, biochemical tests, and serology), Gram stain microscopy, and AST. We calculated the number of laboratories achieving fully correct results in each round and, given the difficulty of conducting some detailed procedures over several days, the proportion achieving ≥ 80% correct results. Thus, we considered a score ≥ 80% in a test category as satisfactory performance. To describe performance trends over time, we generated a 4-point moving trend line (to smooth out fluctuations) based on the median proportion of laboratories achieving satisfactory scores in each round. Similarly, we determined whether laboratories performed better over time on organisms included ≥ 2 times, by looking at the median proportion of laboratories achieving a satisfactory score on repeat specimens. Then we assessed whether any laboratories failed to achieve satisfactory scores in the entire evaluation period by identifying those with a cumulative median score of < 80% in any test category.

Results

Study participation

Thirty-five national public health laboratories participated in the regional EQAS in 2011–2019, with an average of 23 participants in each of the 15 rounds (Figures 1–3). The largest number of laboratories (n = 30) participated in the first round of 2016. The number of participating laboratories increased from 18 in 2011 to 24 in 2019. Twenty (91%) of the 22 EMR countries participated during the study period, with an average of 16 per round. Four countries have not participated since 2016 (data not shown).

Performance in the test categories

The median proportion of laboratories with satisfactory scores remained at 70% for bacterial identification and AST (Figures 1 and 3 trend lines). However, this proportion increased from 80% to 85% for Gram stain microscopy (Figure 2 trend line). The 2016 round 1 had the lowest proportion of laboratories with fully correct results for identification (16/30, 53%) (Figure 1) and Gram stain microscopy (19/30, 63%) (Figure 2). In contrast, the highest proportion of laboratories attained fully correct results in identification (23/24, 96%) and Gram stain microscopy (24/24, 100%) in round 1 of 2019. The 2013 round 1 was the most difficult for participants in the AST category, with just 7 out of 23 laboratories (30%) attaining fully correct results (Figure 3). The most successful round for AST was 2015 round 1 (20/22, 91% with fully correct results).

Performance on specific organisms

We could not establish whether laboratories improved for detection of organisms included ≥ 2 times. However, laboratories consistently identified some organisms more easily than others. For example, Klebsiella pneumoniae (included 3 times) was correctly identified by a median of 90% of laboratories and Yersinia enterocolitica (included 4 times) by 89% (Table 1). In contrast, only 67% of laboratories had satisfactory results identifying normal enteric flora (included 3 times), and 65% correctly identified Salmonella spp., despite the genus being included 10 times in 15 rounds. There was a similar situation for organisms included only once in 2011–2019 (Table 1). There was no clear improvement over time for accuracy of Gram stain microscopy (data not shown), although scores were consistently higher in this than the other test categories (Figure 2). There was no substantial improvement in performance for AST over time, but repeat sample sizes were small and specimens comprised only a small proportion of each round of the regional EQAS. Although Staphylococcus aureus was included for AST 5 times in the study period, only 65% of laboratories achieved satisfactory results (Table 1).

Laboratories not achieving satisfactory scores

Over the study period, 10/35 (29%) laboratories failed to attain a cumulative median score ≥ 80% in at least 1 task category. Seven of these had problems in > 1 category, and 4 had difficulty in all categories. The most frequent task category with low scores was identification (9/10 laboratories), followed by Gram stain microscopy, and antimicrobial susceptibility testing (both 6/10 laboratories).

Discussion

All EMR Member States were represented in the regional EQAS, except Djibouti and the Occupied Palestinian Territories. Participation of national laboratories in the scheme increased between 2011 and 2019. The peak participation was 30 laboratories in the 2016 round 1 but the overall average was 23. This peak resulted from the addition of 4 new laboratories (geographically large countries enrol subregional laboratories) and the return of 2 others. Participation is voluntary and initiated through request to WHO, and there is no agreement binding laboratories to participate. Criteria for the removal of laboratories is also not defined. The 4 countries that did not participate since 2016 were all affected by conflict and had difficulty securing reagents and receiving the test panels from the provider.

Results in the 3 categories of the regional EQAS indicated that 70% of laboratories maintained satisfactory performance in identification and AST, and 85% performed satisfactorily for Gram stain microscopy. Identification and AST are more complicated processes and sometimes take several days. They require quality control procedures and reagents at each step, which leave more room for error. In contrast, Gram stain microscopy requires a simple technique, description of the specimen content and morphology (e.g. white blood cells or Gram-negative bacilli) and sometimes the naming of the pathogen. Nevertheless, the degree of difficulty should not deter laboratories from taking corrective actions after attaining low scores in the scheme. The consequence of poor test results can range from incorrect diagnosis of patient specimens to delay in resolution of disease outbreaks and low confidence in data shared by countries. The lack of an expected performance increase over time in the scheme is comparable to the WHO EQAS in bacteriology in the African Region, which showed that national public health laboratories failed to demonstrate an upward trend in performance over time (8). These observations show that participation in EQASs alone does not improve performance if there are no corrective actions by laboratories.

In some rounds of the EQAS, a low proportion of laboratories attained correct results, particularly in 2016 round 1 for identification and Gram stain microscopy. To explain these results, we examined the organisms administered in those rounds. Fewer than 61% of laboratories could identify Sphingomonas paucimobilis in 2016 round 1, the first time it was included in the scheme. For Gram stain microscopy in 2016 round 1, ≤ 63% of laboratories attained correct results for Campylobacter spp. and Neisseria gonorrhoeae, which were included for the first time in that round. Similarly, the low proportion of laboratories (≤ 65%) attaining correct results for AST in the 2013 round 1 may have been attributable to their performing the test for the first time on Haemophilus influenzae and S. aureus. First encounters, however, were not the sole reason for low scores. When we examined improved testing on organisms included multiple times, some organisms were consistently challenging. This challenge also depended upon which test category they appeared. For example, laboratories had difficulty identifying Shigella boydii and identifying and performing AST on H. influenzae each time they appeared. Identification and Gram stain microscopy of S. aureus was easy for most laboratories each time, and ≥ 96% had satisfactory results in both categories over 5 rounds. However, AST scores were consistently low. The challenge of identifying Salmonella spp. by traditional serotyping methods because of the reagents and technical expertise required (9) was reflected in the scoring for this organism over 10 rounds. Finally, the difficulty that laboratories had in identifying normal enteric flora may have been due to participants’ bias in expecting to encounter pathogens in their test panels. These findings suggest problem test categories and organisms that could benefit from more attention. The WHO EQAS in bacteriology in the African Region

similarly demonstrated that AST was a more challenging test category and more difficult to perform on specific pathogens, such as H. influenzae (10). Performance of AST is of particular concern given the public health importance of antimicrobial resistance. The Global Antimicrobial Resistance Surveillance and Use System (GLASS), under the global action plan to tackle antimicrobial resistance, relies on high-quality surveillance data from countries to assess the burden of resistance, monitor international spread of resistance, and help inform evidence-based policy (11, 12). Inclusion of 1 or 2 specimens per round for AST has long been a part of the regional EQAS. However, only 3 of the 8 priority pathogens in GLASS (13) have ever been included for AST (S. aureus, K. pneumoniae, and Escherichia coli).

Most laboratories achieved satisfactory median performance scores over the study period; however, about one-third failed to do so, indicating that they had unresolved quality issues in 1 or more test categories. The identification category was the most problematic for participants, which could be explained by several factors. First, there could have been inherent complexity in the multiple biochemical and other descriptive tests involved, particularly for laboratories not using automated methods. Second, the variety of organisms included may have been a source of difficulty. Compared with other EQASs focused on variations of a single organism (e.g. influenza virus) (3), or a selection of related organisms (e.g. arboviruses of public health concern) (2), identifying a constantly changing selection of organisms in each panel of the regional EQAS is more challenging.

This study had several limitations. We drew conclusions about laboratory performance based solely on results submitted to a bacteriology testing scheme. However, that does not mean that a laboratory that scored poorly here could not excel in another discipline. Laboratories are usually enrolled in other EQASs, such as for serology or molecular testing, which are more automated, involve less hands-on time, and therefore generate less error (although automated systems also require adequate quality control for good results). Our analysis of aggregate data precluded a more detailed investigation of the patterns that we observed and we did not seek input from participating laboratories on these patterns. To prove our hypothesis that some pathogens were intrinsically difficult to identify would likely

require a more systematic approach, with laboratories testing difficult versus easy pathogens over a period of time. Thus, our work cannot be considered a definitive study on quality testing in bacteriology in regional laboratories. The study should also not be used to define the overall quality of specific laboratories, which perform other operations besides bacteriological testing and may have been affected by a variety of factors, such as staffing, quality and availability of diagnostic methods and reagents, and financial resources.

Despite the attention paid to virology and increasing laboratory capacity for detection of viruses with epidemic potential, such as influenza (14), bacterial infections remain a threat to human health (15,16). The rising incidence of antimicrobial resistance and the necessity for accurate characterization of bacterial pathogens to inform surveillance, treatment, and evidence-based interventions reinforce bacteriology as a key area of laboratory work. The aim of the regional EQAS in bacteriology was to provide laboratories with an opportunity to troubleshoot testing issues, address gaps in quality, and increase their performance over time. However, performance trends remained unchanged, certain test categories (identification and AST) and pathogens consistently resulted in low scores, and there was a minority of laboratories that always produced poor results. Corrective actions in an EQAS are primarily the responsibility of the participants, and range from conducting root cause analysis of problems to ensuring that standard laboratory methods are followed and the necessary reagents are available (1).

Our study demonstrates that action is needed in the form of increased oversight by WHO and the provider to better deliver on the expectations of Member States from the scheme. Such oversight includes the creation of a technical advisory group (17) to critically review the performance of participants and plan rounds, coordinate targeted assistance, align the regional EQAS with global priorities (e.g. GLASS), and increase accountability through the publication of results. We anticipate that a more active approach will help WHO generate the intended value and outcome for laboratories participating in the scheme in bacteriology.

Funding: None

Competing interests: None declared.

Évaluation externe de la qualité des performances des laboratoires en bactériologie dans la Région de la Méditerranée orientale, 2011-2019

Résumé

Contexte : Depuis 2007, les laboratoires de santé publique nationaux de la Région OMS de la Méditerranée orientale ont participé à un système régional d'évaluation externe de la qualité en bactériologie afin d'améliorer la bonne exécution des analyses.

Objectifs : Évaluer les performances des laboratoires en bactériologie dans la Région de la Méditerranée orientale entre 2011 et 2019 à l'aide du système régional d'évaluation externe de la qualité.

Méthodes : Nous avons analysé l'exactitude des données communiquées par les participants concernant l'identification bactérienne, la microscopie après coloration de Gram et les tests de sensibilité aux antimicrobiens. Pour chaque catégorie, nous avons évalué la performance au fil du temps, la performance sur plusieurs

micro-organismes et avons vérifié si un laboratoire n'a pas obtenu des résultats satisfaisants à plusieurs reprises.

Résultats : Entre 2011 et 2019, 70 % des laboratoires ont obtenu des résultats satisfaisants pour l'identification bactérienne et les tests de sensibilité aux antimicrobiens, et 85 % ont effectué une microscopie après coloration de Gram satisfaisante. Les tests ne se sont pas améliorés sur plusieurs micro-organismes et les résultats étaient systématiquement faibles pour certains agents pathogènes et certaines catégories de tests. Vingt-neuf pour cent des laboratoires ont eu des résultats insuffisants tout au long de la période d'étude.

Conclusion : Les performances inchangées au cours du temps et les résultats insuffisants des laboratoires soulignent la nécessité d'améliorer le système régional d'évaluation externe de la qualité. Les laboratoires participants et l'OMS doivent collaborer plus activement pour renforcer les domaines qui posent problème.

التقييم الخارجي لجودة أداء المختبرات المتخصصة في علم الجراثيم في إقليم شرق المتوسط، 2011-2019

همايون أصغر، كارين نهابتيان، أماني غنيم، فرانك كونينجز، أمينة الجرداني

الخلاصة

الخلفية: منذ عام 2007، شاركت مختبرات الصحة العامة الوطنية في إقليم شرق المتوسط في مخطط إقليمي للتقييم الخارجي للجودة في علم الجراثيم لتحسين كفاءة الاختبارات.

الأهداف: هدفت هذه الدراسة الى تقييم أداء المختبرات في مجال علم الجراثيم في إقليم شرق المتوسط بين عامَي 2011 و2019، باستخدام النظام الإقليمي للتقييم الخارجي للجودة.

طرق البحث: حللنا دقة البيانات التي أبلغ بها المشاركون فيما يخص تحديد الجراثيم، والفحص المجهري لصبغة جرام، واختبار الحساسية لمضادات الميكروبات. وبالنسبة إلى كل فئة، قيمنا الأداء مع مرور الزمن، والأداء على كائنات حية متعددة، وما إذا كان المختبر قد فشل مرارًا في تحقيق نتائج مُرضية.

النتائج: بين عامَي 2011 و2019، حقق 70% من المختبرات أداءً مُرضيًا في التعرف على الجراثيم وتحديدها واختبار الحساسية لمضادات الميكروبات، ونفذ 85% منها الفحصَ المجهري لصبغة جرام تنفيذًا مُرضيًا. ولم يتحسن مستوى الاختبار على كائنات حية متعددة، وكانت النتائج منخفضة باستمرار فيما يخص بعض مسببات الأمراض وفئات الاختبارات. وكان أداء 29% من المختبرات دون المستوى خلال فترة الدراسة.

الاستنتاجات: إن عدم تغيُّر الأداء بمرور الوقت والأداء القاصر للمختبرات يبرزان الحاجة إلى إدخال تحسينات على الخطة الإقليمية للتقييم الخارجي للجودة. لذا، يتعيَّن على المختبرات المشارِكة و المختبرات المشاركة ومنظمة الصحة العالمية بحاجة إلى العمل بشكل أكثر نشاطا لتقوية مناطق المشاكل.

References

1. Laboratory quality management system: handbook. Geneva: World Health Organization; 2011 (https://www.who.int/publications/i/item/9789241548274, accessed 17 July 2021).
2. Abdad MY, Squires RC, Cognat S, Oxenford CJ, Konings F. External quality assessment for arbovirus diagnostics in the World Health Organization Western Pacific Region, 2013-2016: improving laboratory quality over the years. Western Pac Surveill Response J. 2017; 8(3):27–30. https://doi.org/10.5365/wpsar.2017.8.3.001 PMID:29051839
3. Detection of influenza viruses by reverse transcription polymerase chain reaction: WHO external quality assessment programme summary analysis, 2019. Wkly Epidemiol Rec. 2020;95(6):49–60. https://apps.who.int/iris/handle/10665/330809
4. Jackson S, Peret TCT, Ziegler TT, Thornburg NJ, Besselaar T, Broorn S, et al. Results from the WHO external quality assessment for the respiratory syncytial virus pilot, 2016-17. Influenza Other Respir Viruses. 2020 Nov;14(6):671–7. https://doi.org/10.1111/irv.12771 PMID:32730685
5. World Health Organization-coordinated Global Invasive Bacterial Vaccine-Preventable Disease (IB-VPD) and Rotavirus Surveillance Network Meetings. 15th–18th November 2016. Geneva: World Health Organization; 2016 (https://www.who.int/docs/default-source/immunization/vpd_surveillance/lab_networks/ib-vpd/global-rv-ibd-meeting-report-2016.pdf?sfvrsn=953b6fed_2, accessed 25 October 2022).
6. Countries. Cairo: World Health Organization Regional Office for the Eastern Mediterranean; 2022 [website] (http://www.emro.who.int/countries.html, accessed 25 October 2022).
7. Canadian Microbiology Proficiency Testing Program. EQA programs – grading schemes. Vancouver: University of British Columbia; 2021 (http://cmpt.ca/eqa-programs/eqa-programs-grading-schemes/, accessed 25 October 2022).
8. Frean J, Perovic O, Fensham V, McCarthy K, Gottberg AV, Gouveia LD, et al. External quality assessment of national public health laboratories in Africa, 2002-2009. Bull World Health Organ. 2012 Mar 1;90(3):191–9A. https://doi.org/10.2471/BLT.11.091876 PMID:22461714
9. Diep B, Barretto C, Portmann AC, Fournier C, Karczmarek A, Voets G, Li S, et al. Salmonella serotyping; comparison of the traditional method to a microarray-based method and an in silico platform using whole genome sequencing data. Front Microbiol. 2019 Nov 11;10:2554. https://doi.org/10.3389/fmicb.2019.02554 PMID:31781065
10. Perovic O, Yahaya AA, Viljoen C, Ndihokubwayo JB, Smith M, Coulibaly SO. External Quality Assessment of Bacterial Identification and Antimicrobial Susceptibility Testing in African National Public Health Laboratories, 2011-2016. Trop Med Infect Dis. 2019 Dec 13;4(4):144. https://doi.org/10.3390/tropicalmed4040144 PMID:31847247
11. Global action plan on antimicrobial resistance. Geneva: World Health Organization; 2015 (https://www.who.int/publications/i/item/9789241509763, accessed 25 October 2022).
12. Global Antimicrobial Resistance Surveillance System (GLASS). Geneva: World Health Organization [website] (https://www.who.int/initiatives/glass, accessed 25 October 2022).
13. GLASS report: early implementation 2020. Geneva: World Health Organization; 2020 (https://www.who.int/publications/i/item/9789240005587, accessed 9 June 2022).
14. Squires RC, Reading PC, Sullivan SG, Barr IG, Konings F. Influenza virus detection: driving change in public health laboratories in the Western Pacific Region. Western Pac Surveill Response J. 2018 Sep 5;9(5 Suppl 1):68–70. https://doi.org/10.5365/wpsar.2018.9.5.006 PMID:31832255
15. Woolhouse M, Waugh C, Perry MR, Nair H. Global disease burden due to antibiotic resistance – state of the evidence. J Glob Health. 2016 Jun;6(1):010306. https://doi.org/10.7189/jogh.06.010306 PMID:27350872
16. Tuberculosis fact sheet [website]. Geneva: World Health Organization; 2020 (https://www.who.int/news-room/fact-sheets/detail/tuberculosis, accessed 25 October 2022).
17. WHO manual for organizing a national external quality assessment programme for health laboratories and other testing sites. Geneva: World Health Organization; 2016 (https://apps.who.int/iris/handle/10665/250117, accessed 17 July 2021).