Wide Variability in the Sensitivity and Specificity of Rotavirus Immunoassay Diagnostic Kits in Practice

Introduction: Most hospitals rely on rapid antigen-detection kits for the diagnosis of rotavirus infection. Several small studies reviewed the sensitivity and specificity of some of these kits. These studies showed discrepancy in results obtained for sensitivity and specificity that varied according to the type of kit used, area of study, and type of test used as standard for diagnosis of rotavirus infection. The objective of the study is to determine the sensitivity and specificity of five commonly used rotavirus immunoassay kits in comparison to RT-PCR as standard. Methodology: Stool samples (N=1,414) collected from children under 5 years of age hospitalized with gastroenteritis were tested for rotavirus by immunoassay kits and RT-PCR in a prospective hospital-based surveillance study conducted at 7 centers in Lebanon. Concordance and discrepancy between the two methods was used to calculate sensitivity and specificity, using RT-PCR as the “gold standard”. Results: The sensitivity and specificity were respectively 95.08% and 86.62% for the SD Bioline® (Standard Diagnostics, Inc, South Korea) kit calculated on 645 samples, 65.86% and 45.90% for the VIROTECT® (Trinity Biotech, Ireland) kit calculated on 327 samples, 83.9% and 64.2% for the Rota-Strip (C-1001) (Coris Bioconcept, Belgium) calculated on 95 samples, 52.3% and 10.9% for the Acon® (Acon Laboratories, Inc, California, USA) kit calculated on 122 samples, 68.1% and 20% for the VIKIA® Rota-Adéno (Biomerieux, France) kit calculated on 32 samples. Conclusions: A wide discrepancy was detected between the calculated and advertised sensitivity and specificity for most of the kits.


Introduction
Rotavirus (RV) is the most common diarrheal pathogen, accounting for approximately 30-40% of all acute gastroenteritis hospitalizations in both industrialized and less developed countries, with an estimated 450,000 deaths per year in children under 5 years of age [1][2][3][4]. The mortality attributed to RV infection is especially high in developing countries, with more than half of deaths occurring in Africa in the pre-vaccine era [5]. The introduction of human Rotavirus vaccine (HRV) played a major role in the decrease of RV gastroenteritis-associated mortality and hospitalizations in the countries that implemented the vaccine in its national immunization programs [5,6].
Besides, rapid and accurate diagnosis plays a major role in the control of virus transmission and subsequently in disease prevention through implementation of isolation precautions [7].
RV was first identified in 1973 by electron microscopy (EM), which was the main method used for the diagnosis of RV infections before the advent of latex agglutination (LA) tests, enzyme immunoassays (EIA), and reverse transcription-polymerase chain reaction (RT-PCR) [8][9][10][11][12][13]. Currently, the most reliable methods for the diagnosis of RV infection are nucleic acid-based detection techniques; however, their cost, time consumption and requirement for specialized laboratory equipment, limit their use [14]. In practice, most health care facilities use antigen-detection assays to diagnose RV infections [15]. The reported sensitivity and specificity of kits from various manufacturers have varied significantly. In this observational study, we aimed to determine the sensitivity and specificity of commonly used RV detection kits. This would provide essential information on the accuracy of kits used in "real life" conditions rather than under controlled study conditions, and help provide more accurate data on disease prevalence and epidemiology in studies using these kits.

Methodology
In 2013, we conducted a prospective hospital based surveillance study in Lebanon that included seven hospitals from different areas to determine RV disease burden and prevalent genotypes causing gastroenteritis in hospitalized children under 5 years of age. Six different immunoassay kits were used for the diagnosis of RV gastroenteritis at the participating hospitals. All referred stool samples were retested by the SD Bioline (Standard Diagnostics, Inc, South Korea) rotavirus rapid kit at the Center for Infectious Diseases Research (CIDR) at the American University of Beirut Medical Center (AUBMC), where the study was being coordinated. Discrepancy in the results was observed between the referring centers and the central laboratory. Therefore, we decided to determine the sensitivity and specificity of these RV detection kits used in the seven participating hospitals, using RT-PCR as standard for diagnosis. The six used immunoassay kits were: Rotacheck-1® (VEDALAB, France) kit, SD Bioline Rotavirus rapid kit (Standard diagnostics, Inc, South Korea), VIROTECT® (Trinity Biotech, Ireland) kit, Rota-Strip (C-1001) (Coris Bioconcept, Belgium) kit, VIKIA ® Rota-Adéno (Biomerieux, France) kit, Acon® (Acon Laboratories, Inc, California, USA) kit. Subjects were enrolled if they were admitted to one of seven hospitals with the diagnosis of gastroenteritis and had not reached their 5 th birthday. The hospitals were distributed in North Lebanon (Nini Hospital), Central Lebanon (AUBMC, Rafic Hariri University Hospital (RHUH), Makassed General Hospital (MGH), Hotel Dieu De France (HDF), and South Lebanon (Hammoud Hospital University Medical Center (HHUMC), Nabatiyeh Governmental Hospital (NGH). They were excluded if the gastroenteritis onset was more than 12 hours after hospital admission. After obtaining informed consent, a case report form was used to collect information that included demographic data (age, gender, weight), past medical history and symptoms. Information included any previous treatments, vaccination, and area of residence. Stool samples were collected from all enrolled subjects preferably within 4 days and not later than 10 days after the onset of symptoms and processed in accordance with study guidelines. Stool samples were stored in a refrigerator at the participating hospital at a temperature between 2 and 8 degrees Celsius up to 72 hours, and after this period they were transferred to CIDR where they were stored at a temperature between -20 °C and -70 °C until further processed. A total of 1414 subjects were enrolled and their stool samples were collected over a period of 30 months. All the 1414 samples collected from the participating hospitals were retested at CIDR using the SD Bioline Rotavirus rapid kit (Standards Diagnostics, INC., Republic of Korea) according to the manufacturer's instructions. All samples that tested positive either by the kit used at the participating hospital, or by the CIDR kit, or by both kits underwent RNA extraction and gene sequencing. A 10% stool suspension (0.5 g of fecal sample added to 5 ml of NaCl solution (0.89%) was used for RNA extraction. The homogenate was centrifuged at 4,000 g at 4°C for 20 minutes, then the supernatant was re-centrifuged at 1,500 g at 4°C for 10 minutes. The clarified supernatant (420 μl) was used for RNA extraction using the QIAamp ®Viral RNA Mini Kit (Qiagen, Hilden, Germany). The extracted RNA was denatured at 97°C for 5 minutes and reverse transcription followed by polymerase chain reaction (RT-PCR) was carried out using the Qiagen One Step RT-PCR Kit (Qiagen, Germany) with the following thermal cycling conditions: initial reverse transcription step at 42°C for 30 minutes, followed by initial denaturation at 95°C for 15 minutes, 30 cycles of amplification at 94°C for 30 seconds, 50°C and 53°C for the viral proteins (VP) (VP4 and VP7), respectively for 30 seconds, 72°C for 42 seconds and a final extension at 72°C for 7 minutes. The nucleotide (nt) positions and sequences of the primers (5' to 3') used for the amplification of VP4 and VP7 genes were as follows: con 3 (nt 11 to 32), TGGCTTCGCCATTTATAGACA; con 2 (nt 868 to 887), ATTTCGGACCATTTATAACC; 9con1-L (nt 37 to 59), TAGCTCCTTTTAATGTATGGTAT; VP7 (nt 914 to 933) AACTTGCCACCATTTTTTCC as previously published [16].
Each gene was amplified on a C1000 thermal cycler (Bio-Rad, Inc., Berkeley, California, USA). Amplicons were detected by gel electrophoresis and analyzed by a gel documentation system (Gel doc XR, Bio-Rad, Berkeley, California, USA). RT-PCR products were cleaned using ExoSAP-IT ® (USB Corp., Cleveland, OH, USA), then sanger sequenced at Macrogen Institution (Seoul, Republic of Korea). Nucleotide sequences were assembled using ClustalW tool included in BioEdit Sequence Alignment Editor v7.2.5. The corresponding G and P genotypes were assigned by RotaC v2.0 software (http://rotac.regatools.be/) for automated genotyping of RVA and confirmed using the Basic Local Alignment Search Tool (BLAST) available on GenBank database (http://blast.ncbi.nlm.nih.gov/) [17]. For those stool samples that tested negative by rapid kits at both the referring hospital and at the CIDR (neg/neg), the possibility that a rotavirus infection was missed by both kits due to false-negative results was explored by testing every 20 th neg/neg sample by RT-PCR. Only RT-PCR positive samples were considered as true positives. The sensitivity (Sn) and specificity (Sp) of the different kits were then calculated in comparison with the RT-PCR results. Samples that were tested positive by the referring kit and by RT-PCR were considered as true positive (TP), those that tested positive by the referring kit and negative by RT-PCR were considered as false positive (FP), those that tested negative by the referring kit and by RT-PCR were considered as true negative (TN) and those that tested negative by the referring kit and positive by RT-PCR were considered as false negative (FN). Sn was calculated by dividing TP over TP plus FN, Sp was calculated by dividing TN over TN plus FP, positive predictive value (PPV) was calculated by dividing TP over TP plus FP, and negative predictive value (NPV) was calculated by dividing TN over TN plus FN. Table  1 summarizes the different types of RV antigen detection kits used in the participating hospitals, the advertised sensitivity and specificity, and the method used for detection.

Results
A total of 1414 samples were collected, 1361 samples were tested at the referring hospitals and retested at CIDR, and the rest (53 samples) were only tested at the CIDR. The number of samples tested concomitantly by the corresponding kit and by RT-PCR was 584 for the SD Bioline (Standard Diagnostics, Inc, South Korea) kit, 269 for the VIROTECT® (Trinity Biotech, Ireland) kit, 120 for the Acon® (Acon Laboratories, Inc, California, USA) kit, 95 for the Rota-Strip (C-1001) (Coris Bioconcept, Belgium), and 32 for the VIKIA® Rota-Adéno (Biomerieux, France) kit (Table 2). Accordingly, the sensitivity and specificity of these kits were calculated. The results from HDF were not included in the analysis because only 5 samples were submitted by this center, and no samples were tested by RT-PCR, so the analysis of Rota-check-1 ® was not included.
The SD Bioline® (Standard Diagnostics, Inc, South Korea) kit used at the CIDR and the Rota-Strip (C-1001) (Coris Bioconcept, Belgium) kit used at HHUMC performed best showing a sensitivity of 95.08% and 83.9%, and a specificity of 86.62% and 64.2%, respectively. For the SD Bioline (Standard Diagnostics, Inc, South Korea) kit, the calculated sensitivity was approximately similar to that advertised by the manufacturer of 94%; however, the calculated specificity was below the advertised value of 98.3%. For the Rota-Strip (C-1001) (Coris Bioconcept, Belgium) kit, the calculated sensitivity and specificity were below that advertised by the manufacturer of 97.3% and 97%, respectively ( Table 1). The VIROTECT® (Trinity Biotech, Ireland) kit used at MGH and Nini Hospital showed a sensitivity of 65.86%, significantly lower than that reported by the manufacturer of 97%, a specificity of 45.9% also far below the value advertised by the manufacturer of 97%. The calculated sensitivity and specificity were respectively 68.1% and 20% for the VIKIA ® Rota-Adéno (Biomerieux, France) kit used in RHUH, 52.3% and 10.9% for the Acon® (Acon Laboratories, Inc, California, USA) kit used in NGH; however, the advertised sensitivity and specificity were drastically higher, above 96.3% and 98.6%, respectively for the VIKIA ® Rota-Adéno (Biomerieux, France) kit, and above 97.9% and 93.6%, respectively for the Acon® (Acon Laboratories, Inc, California, USA) kit.

Discussion
The observed results showed significant discrepancy between the calculated sensitivity and specificity and that advertised by the manufacturers for most of the kits used in this study. The VIROTECT® (Trinity Biotech, Ireland) kit used at MGH and Nini Hospital is the only kit relying on the LA test in the current study for the rapid detection of rotavirus antigen. The calculated sensitivity and specificity of 65.86% and 45.9%, respectively, were significantly lower than the values advertised by the manufacturer of 97.2% and 97.1%, respectively, and that reported for other LA tests [18][19][20][21]. In a study published by Bon F et al., the sensitivity of the Slidex Rota/Adeno (BioMérieux, France) LA test was 77.5% in 80 ELISA positive fresh stool samples. In 100 negative fresh stool samples, the specificity of the test was 100% [18]. Another study that evaluated the Slidex Rota/Adeno (BioMérieux, Marcy-l'Etoile, France) kit reported a sensitivity of 82% and a specificity of 100%, as compared to EM and ELISA [19]. The Rotalex (Orion Diagnostica of Helsinki, Finland) test, another commercial LA test for rotavirus detection, was reported to have a sensitivity of 81.7% and a specificity of 99.5% as compared to EM [21]. Raboni et al. evaluated the LA Rotagen (Biokit S.A., Barcelona, Spain) kit for the detection of rotavirus antigen in 285 fecal samples. The reported sensitivity and specificity, as compared to EIA, were 69% and 100%, respectively [12]. The discrepancies between the obtained results, those advertised by the manufacturer and those reported in the literature for similar LA tests might be attributed to two main reasons. First, the fact that the previous studies used EM and ELISA as standards for diagnosis and the current study used RT-PCR, which is more sensitive and specific than non molecular methods [13], could explain the low calculated sensitivity. A study conducted by Gautam et al. revealed that the sensitivity of three commercially available EIA ranged between 75% and 82.1% as compared to RT-PCR [22]. Second, the sensitivity of the LA tests was shown to correlate with the stage of the disease at which specimens are collected. The sensitivity of a commercial LA kit, the Rotalex (Orion Diagnostica of Helsinki, Finland), was 100% during the first 4 days of illness and 96% between the fifth and seventh day of illness, and then it decreased significantly thereafter. The concentration of virus  particles shed in stools is highest in the first week of illness [21]. The low sensitivity in the current study could be related to the fact that these samples were collected at a later stage of the illness when viral shedding significantly decreased. The calculated PPV of 80.58% was within the values reported in the literature ranging between 76% and 100%; however the calculated NPV of 28.28% was lower than the values reported in the literature ranging between 85% and 100% [21,23]. Agglutination is seen by the naked eye and the final result is operator-dependent, leading to false negative results according to the experience of the laboratory technician [24], thus affecting the NPV. All the remaining kits used in the current study (Rota-Strip (C-1001) (Coris Bioconcept, Belgium), VIKIA® Rota-Adéno (Biomerieux, France), SD Bioline® (Standard diagnostics, Inc, South Korea), Acon® (Acon Laboratories, Inc, California, USA) were rapid immunochromatographic assays for the detection of rotavirus antigen in stool.
The Rota-Strip (C-1001) (Coris Bioconcept, Belgium) kit used at HHUMC revealed a sensitivity and specificity of 83.9% and 64.2%, respectively, that are lower than the advertised values of 97.3% and 97.7%, respectively, and those reported in the literature [18]. The sensitivity and specificity of the Rota-Strip (C-1001) evaluated by Bon F et al. were reported to be high of 98.8% and 100%, respectively; however, this study used an EIA (Argene) test as a reference method for the diagnosis of group A rotavirus in fecal samples [18].
The VIKIA® Rota-Adéno (Biomerieux, France) kit used at RHUH for the simultaneous rapid detection of rotaviruses and adenoviruses revealed a sensitivity and specificity of 68.1% and 20%, respectively, far below the ranges advertised by the manufacturer ranging between 96.3% and 100%, 98.6% and 100%, respectively. The calculated values were also lower than that reported by other studies that either used ELISA [18], or genotyping as standards for diagnosis [25]. A study published by Bon F et al. evaluated seven immunochromatographic assays, including the VIKIA® Rota-Adéno (Biomerieux, France) kit, for the rapid detection of human RV in fecal specimens as compared to an EIA (Argene). In 80 ELISA-positive frozen stool samples, human RV was detected in 92.5% by the VIKIA® Rota-Adéno (Biomerieux, France). No false positive results were detected in 100 fresh rotavirus negative stools [18]. De Rougemont et al. also evaluated the sensitivity and specificity of the VIKIA ® Rota-Adéno test using 57 stool samples that were compared to genotyping as a referral method. The sensitivity and specificity were high and comparable of 96.6% and 96.4%, respectively [25]. Another study published by Ye et al. in 2013 showed a lower specificity for the VIKIA ® Rota-Adéno (Biomerieux, France) kit in the vaccine era in Australia. It tested 133 fecal samples and showed that only 28-37% of samples positive by the VIKIA ® Rota-Adéno (Biomerieux, France) kit could be confirmed using two real-time RT-PCR assays and three ELISA kits [26]. This was mostly attributed to the low disease incidence in a high vaccination coverage setting. A recent study published by Kaplon et al. in 2015 that evaluated seven immunochromatographic assays including the VIKIA ® Rota-Adéno (Biomerieux, France) kit reported a sensitivity of 77.3% and interestingly a 100% specificity [27]. The test was performed on 253 stool samples and was compared to RT-PCR as a referral method.
The SD Bioline® (Standard diagnostics, Inc, South Korea) used at the CIDR at AUBMC revealed a sensitivity of 95.08%, similar to that reported by the manufacturer of 94% and that reported in the literature [28]. The calculated specificity of 86.62% was lower than that reported by the manufacturer of 98.3% and by another study that assessed the performance of the SD Bioline rota/adeno (Standard diagnostics, Inc, Kyonggi, Korea) kit as compared to RT-PCR [28]. The latter study evaluated 755 stool samples collected from children with acute diarrhea. The SD Bioline rota/adeno (Standard diagnostics, Inc, Kyonggi, Korea) kit was found to have a sensitivity of 93.5% and a specificity of 96.1% [28]. However, another study published by Kaplon et al. evaluated 253 stool samples and reported a lower sensitivity of 77.3% and a higher specificity of 97.9% as compared to RT-PCR [27].
Finally, the calculated sensitivity and specificity of the Acon® (Acon Laboratories, Inc, California, USA) kit used at NGH were 52.3% and 10.9%, respectively, also far below that advertised by the manufacturer ranging between 97.9% and 100%, 93.6% and 100%, respectively.
There were no available studies in the literature, reporting the sensitivity and specificity of the Acon (Acon Laboratories, Inc, California, USA) kit. This study has limitations as it was not initially designed to look at specificity and sensitivity of RV detection kits and the number of samples tested by some of the kits was small. Stool samples were stored in a refrigerator at the participating hospital at a temperature between 2 and 8 degrees Celsius up to 72 hours, and after this period they were transferred to CIDR where they were stored at a temperature between -20°C and -70°C until further processed. Although rotavirus is known to be a hardy virus [29][30][31], the storage may lead to degradation of the virus and decreases the specificity of the test leading to an apparent false positive initial read. Also, samples were handled by different operators before being transported to the central laboratory at CIDR leaving the possibility of contamination along the process, decreasing the sensitivity due to an apparent false negative initial read. Different operators used the different kits, even at the same hospital, allowing for operator-dependent variation. However, this is in fact what happens in real life.

Conclusions
Reliable detection of RV in patients with diarrhea is important for diagnosis, management, and inpatient decisions regarding isolation. Rapid antigen detection kits for RV are convenient, inexpensive, and available in most hospitals caring for children. Based on our study, not all of these kits perform similarly. The advertised specificity and sensitivity were not observed in our "real world" study. The discrepancy between the calculated and the reported results for all used kits may indicate that there is significant "operator-dependence" that becomes evident when these kits are evaluated in a real life situation versus a controlled study setting. Manufacturers of RV kits, as well as other types of kits, should take this important factor into consideration by manufacturing kits that are as operator-independent as possible. As RT-PCR testing becomes more readily available, less expensive, and with shortened turnaround time, its use is likely to replace rapid detection kits especially for research studies where measuring the true burden of disease is desirable.

Funding
The initial work (15) was funded by Merck Sharp & Dohme (MSD) (IISP ID 38155). MSD was not involved in any aspect of the currently reported study.