Comparison of two commercial multiplex PCR assays for the detection of sexually transmitted infections

Introduction: Multiplex molecular panels are replacing conventional methods for the detection of sexually transmitted infections. In the current study, we evaluated the performance of two commercial multiplex assays, EUROArray STI and Allplex STI essential assays, for detecting six sexually transmitted infections. Methodology: The diagnostic performance of the EUROArray STI and Allplex STI essential assays was evaluated against a panel of 105 positive DNA samples identified by in-house real-time PCR assays including Ureaplasma parvum, Ureaplasma urealyticum, Mycoplasma hominis, Trichomonas vaginalis, Chlamydia trachomatis, and Neisseria gonorrhea. Samples from healthy subjects, negative for any microorganism, were used as negative controls. Results: Of the 105 positive specimens, 103 (98%) were tested positive by Allplex and 102 (97%) by EUROArray. Among the 51 negative samples that were tested by in house assay, 48 (94%) were tested negative by Allplex assay and 43 (84%) by EUROArray assay. The overall sensitivity of EUROArray and Allplex were 97.1% and 98.1% with an accuracy of 92.9% and 96.7%, respectively. The overall assay specificity was 94.1% for Allplex assay and 84.3% for EUROArray assay, The sensitivity of both kits to all targeted microorganisms ranged from 55.6% to 100%, with the lowest sensitivity noted for Trichomonas vaginalis. Conclusions: Diagnostic performance varies depending on the method used to detect the targeted pathogens, the assay manipulation, and the cost. This study showed sensitivity, specificity, and accuracy characteristics for two kits commonly used to detect STIs, which will guide the choice for an appropriate multiplex PCR platform.


Introduction
Sexually transmitted infections (STI) have a welldocumented impact on the public health sector since they cause medical, social and economic sequelae [1][2][3][4][5]. The importance of STI lies in the fact that they can cause complications in infected individuals [4,6]. Unspecific symptoms and subclinical infections may lead to challenges in the clinical diagnosis. Laboratoryconfirmed etiological diagnosis is therefore the most reliable approach that should be used in the management of such infections [7,8].
Several pathogens may cause STI ranging from bacteria, viruses, fungi and parasites; conventionally, various diagnostic assays were used to identify them including wet mount, gram stain, cell culture and serological assays [9][10][11][12][13][14][15][16][17][18] (Supplementary Table 1). However, some of the fastidious microbes may not be easily identified by any of those methods such as Mycoplasma and Ureaplasma species. In the last decades, these techniques were replaced by nucleic acid amplification tests (NAAT) [19][20][21][22][23][24], which are currently recommended for screening and clinical diagnosis; they proved to be more sensitive and more specific than conventional assays.
The new era of molecular diagnostics is not only expected to accelerate detection but to also replace traditional methods and enter all disciplines and diagnostic fields. As demonstrated in the ongoing improvements of the new syndromic panel-based platforms to enhance assays, save time, make specimen preparation easier, and utilize multiplex platforms for the simultaneous detection of pathogens along with automation [25].
Several commercial assays or systems based on nucleic acid amplification have been developed to increase the sensitivity and specificity to detect simultaneously the most prevalent sexually transmitted pathogens [26][27][28].
The aim of the study was to evaluate the performance of two commercial multiplex assays, EUROArray STI and Allplex STI essential assays, for the detection of six pathogens: Chlamydia trachomatis (CT), Neisseria gonorrhoeae (NG), Trichomonas vaginalis (TV), Ureaplasma urealyticum (UU), Ureaplasma parvum (UP), Mycoplasma hominis (MH), and Mycoplasma genitalium (MG). This unprecedented comparison will provide essential information for a better choice of diagnostic tools.

Study specimens
Two genital flocking swabs (endocervical and vaginal) were collected by healthcare practitioners from women seeking gynecological checkup (N = 505) for a period of one year and placed in a single tube containing universal transport media (Copan Diagnostics Inc, Murietta, USA). DNA from 200 µL of each sample was extracted using a QIAamp DNA extraction kit (Qiagen, Hilden, Germany) according to the manufacturer's protocol. The samples were tested by an in-house multiplex real time PCR assay and a diagnosis was given accordingly [29]. For cost limits, only 105 positive samples for Chlamydia trachomatis (CT), Neisseria gonorrhea (NG), Mycoplasma genitalium (MG), Ureaplasma urealyticum (UU), Urealplasma parvum (UP), Trichomonas vaginalis (TV), and Mycoplasma hominis (MH) were selected for further testing by commercial assays along with 51 negative specimens.

STI Commercial detection assays
Two commercial assays were used for sensitivity/specificity analysis: the commercial multiplex real time PCR assay (Allplex STI essential assay, Seegene, Seoul, Korea) and the commercial fluorescence-based DNA microarray assay ( Both assays were performed according to the manufacturer's protocol using the CFX96 real time thermocycler (Bio-Rad-Hercules, CA, USA). A second step is however required for the EUROArray STI to detect the amplified products using an oligonucleotide DNA probe ship which is then read on a EUROArrayScan software (EUROIMMUN).

Statistical analysis
In house PCR was considered as the gold standard procedure for testing the investigated pathogens upon which the two commercial kits were compared.
The sensitivity and specificity of each test and for each microorganism was calculated. The positive predictive value (PPV) and negative predictive value (NPV) were calculated for all specimens using JMP®, Version 15. SAS Institute Inc., Cary, NC, 1989-2019. The threshold for significance was set at 0.05. This study was approved by the Institutional Review Board of the Saint Georges Hospital University Medical Center (Number: 009, Date: 2016).

Results
Of the 105 positive specimens, 103 (98%) were positive by Allplex and 102 (97%) by EUROArray. Among the 51 negative samples by in house assay, 48 (94%) were shown to be negative by Allplex and 43 (84%) by EUROArray. The overall performance of the two assays was compared for their accuracy, sensitivity, specificity, PPV and NPV ( Table 1).
The overall sensitivity of EUROArray and Allplex were 97.1% and 98.1% with an accuracy of 92.9% and 96.7%, respectively. As for the specificity, Allplex Seegene was seen to be more specific than EUROArray, 94.1% versus 84.3%, respectively.
The positive predictive value (PPV) and negative predictive value (NPV) were slightly higher in Allplex compared to EUROArray reaching a PPV of 97% and 93% and a NPV of 96% and 94% respectively. No significant differences in the overall performance of the two assays was detected.
We then calculated the inter-rater reliability, the Cohen's kappa coefficient ( ), for both tests. Our results showed a very good agreement in the two assays when testing for the 6 microorganisms ( Table 1).
The performance of both assays for six targets was compared: UU, UP, CT, TV, MH, MG and NG. The overall sensitivity of both kits to all targeted microorganisms ranged from the lowest rate of 55.6% to 100% (Table 1). Both assays exhibited a poor sensitivity for TV. High number of false positives were detected in EUROArray for MH (N = 9, PPV = 64%), TV (N = 3, PPV = 62.5%), CT (N = 2, PPV = 60%), NG and MG (N = 1, PPV = 50%), which is likely attributed to the low PPV of the tested microorganism. The PPV of MH was also low (57.1%) when tested by Allplex assay. The overall specificity of Allplex assay was high when detecting the targeted microorganisms ranging from 79% to 100%. The accuracy of the assays targeting the microorganisms ranged from 82.3% to 100% in both kits (Table 1).

Discussion
In the last decades, new diagnostic molecular tools have been implemented in diagnostic laboratories including monoplex PCR, multiplex PCR and real time PCR assays [21][22][23][24]27,28,[30][31][32]. For the application of these assays, various factors are usually considered including hands on time, targets coverage, sensitivity, specificity, degree of automation, and cost. These assays are now considered the test of choice for the diagnosis of infection; they replaced the need for traditional testing methods, improved diagnostic performance and can now detect the presence of multiple organisms in one tube [19].
Multiplex assays that target more than three organisms causing STI's in one assay have been developed commercially. The currently available CE market multiplex assays include: FilmArray STD Panel (BioFire Diagnostics), FTD STD9 (Fast-track Diagnostics), Allplex™ STI Essential (Seegene), STD finder (PathoFinder), STI EUROArray (Euroimmun), STI multiplex array (Randox Biosciences) and VIASURE sexually transmitted diseases (CerTest Biotec). With the increase availability of assays, comparison between commercial kits should be performed to assess their performance including their workflow. Therefore, the current study attempted to evaluate the performance of two commercial assays in terms of sensitivity, specificity and accuracy. Table 2 shows the workflow analysis of Allplex™ STI Essential and STI EUROArray assays when compared for handson time, turnaround time, number of steps, number of target cost, along with other variables. Both assays were sensitive, however, Allplex achieved the best performance in term of specificity, accuracy and efficiency in many aspects ranging from turnaround time, number of step and simultaneous detection of samples. EUROArray can only test 25 samples at a time while Allplex assay may detect 96 samples in one run and require one instrument. EUROArray was found to be a more technically sensitive since the assay is light, heat and bubble sensitive. However, the turnaround time of the EUROArray is longer and needs 5 hours to complete the test. It also requires expertise in assay mixing and application along with special care to avoid any laboratory contamination. Nonetheless, the turnaround time of both assays does not have an impact on patient's treatment as the results are provided on the same day.
Allplex STI is one step assay; it reduces hands on time and prevents possible contamination along with having semi-quantitative result while EUROArray provides qualitative results. The Allplex STI assay detects a single channel multiple fluorophores values while EUROArray PCR products are hybridized to biochip microarray slides containing immobilized complementary DNA probes. The positivity of the samples is detected by their fluorescence signals read on an additional scanner instrument required in the laboratories.
When the three assays including the in-house were compared for their sensitivity, specificity and accuracy to detect 6 pathogens, both commercial assays had poor sensitivity to detected TV. The low specificity and PPV witnessed for MH in both kits may be due to the probes used in the assays that are affecting the detection of the microorganism. Both assays performed similarly and efficiently in detecting UU and UP.
The study had few limitations including the limited number of patients infected with MH (N = 17), CT (N = 3), TV (N = 9), NG (N = 1) and MG (N = 1) which resulted in a wider confidence interval and prevented meaningful calculations to compare the performance of the two assays in the detection of these microorganisms. In addition, we did not assess the performance of additional targets in the study such as microorganisms causing genital ulcer due to the absence of positive samples.
In conclusion, it is recommended that before the use of any routine STI diagnostic platform, comparative analysis should be done. Various factors should be evaluated, the sensitivity and specificity of the assay to detect various microorganisms, the assay manipulation, and the cost that may vary between countries. Additional studies are also needed to compare the performance of all assays available in the market.

Author's Contributions
MEC supervised and analyzed the experiments and was responsible for writing up the manuscript. RY and JH performed and analyzed the experiments in addition to data analysis. RY and JH have equal contribution. REB helped in the statistical analysis of the data. JM helped in the writing up of the manuscript. All authors reviewed the manuscript. Some microscopic findings can be misinterpreted due to lack of experience, skills, and credibility. No complete picture of universal standards in diagnosis. Lack of objectivity in testing. Not routinely tested in laboratories. Needs special agar medium and long incubation time.

Annex -Supplementary items
Varying sensitivity. Low specificity.

Neisseria gonorrhea
Gram-negative diplococci under Gram-stain. Culture on modified Thayer-Martin medium (selective medium). Identification by glucose and maltose biochemical tests.
Relatively timely and labor intensive. Needs specialized media, specimen handling, collection methods, and transportation conditions. Fastidious pathogen (difficult to grow).

Relatively low
sensitivity. Low specificity. [9][10][11][12] Chlamydia Direct sexual contact. During birth. False positive results can happen. Need highly trained cytotechnologists to give the correct interpretation. Need optimal smears to be taken by clinicians for best diagnosis.
Disruption of the normal flora.
Culture on sabouraud and blood agars. Blastoconidia detection under gram stain.
Germ tube test.
Long turnaround time for species identification.
Highly sensitive and specific. [17] Parasitic Trichomoniasis Direct sexual contact. Trichomonas vaginalis Wet mounts to detect Trichomonas under the microscope. InPouch TV Culture System (a kit that combines between culture and wet mount).
TV Culture System is expensive and not readily available in laboratories and timely (identification can need from 2 to 7 days). Wet mount needs high expertise, fast specimen transport, and quick processing since the organism can easily loose motility and viability.
Wet mount has low sensitivity. TV culture system has higher sensitivity than wet mount yet less than PCR techniques. Very high specificity. [18]