The FDA gets aggressive with Covid-19 antibody tests
Chembio’s antibody test is the first to have its authorisation revoked. It might not be the last.
The US FDA’s rescinding of emergency use authorisation for Chembio’s Covid-19 antibody test on grounds of inaccuracy caused the company’s stock to plunge more than 60% yesterday. It might also have prompted the makers of other antibody tests with EUAs to wonder whether the agency's action set a precedent.
The regulator took the action after performing its own investigation into the accuracy of the DPP Covid-19 IgM/IgG system, as Chembio’s test is called. And it is doing the same for the other assays it has authorised. Three so far have been given the all-clear, which leaves 13 authorised antibody tests still potentially vulnerable to being double-checked and found wanting.
The FDA began granting emergency market passes to antibody tests at the start of April, based on accuracy data submitted by the manufacturers; the agency checked this over but did not run separate evaluations of its own.
In mid-April the agency said it would start performing independent validations in concert with other US government agencies, including the National Cancer Institute. This has been a slow process, however, with reports on only four commercially produced antibody tests with EUAs having so far been made available. Chembio’s is the first to have run into trouble.
Crunching the numbers
The data Chembio submitted for its test claimed sensitivity of 93.5%, based on correctly identifying 29 of 31 positive samples, and specificity of 94.4%, getting 118 of 125 negative samples right. (These values are for the combined IgM and IgG antibodies; separate values were also given.) Data from an evaluation sponsored by the NCI and performed at the Frederick National Laboratory for Cancer Research found rather different numbers, summarised below.
| Accuracy data on Chembio's antibody test | ||
|---|---|---|
| Initially submitted by Chembio | NCI evaluation | |
| IgM sensitivity | 77.4% | 57.1% |
| IgM specificity | - | 86.2% |
| IgG sensitivity | 87.1% | 78.6% |
| IgG specificity | - | 91.2% |
| Combined sensitivity | 93.5% | 82.1% |
| Combined specificity | 94.4% | 81.2% |
| Combined PPV | 46.8% | 18.7% |
| Combined NPV | 99.6% | 98.9% |
| PPV and NPV at 5% prevalence. Source: FDA. | ||
In its revocation letter to Chembio the FDA wrote that it was "not reasonable to believe that the test may be effective in detecting antibodies against Sars-CoV-2”, or that its benefits outweighed its risks, including the high rate of false results.
The FDA also stated in the letter that its current thinking on antibody tests was that they should have a minimum combined sensitivity of 90% and a minimum specificity of 95%. For tests that report specifically IgM and IgG, minimum sensitivity values of 90% and 70% for IgG and IgM respectively are called for. There are also other conditions including the number of samples on which these figures should be calculated.
So how do the other antibody tests stack up? Of the four assays that have been independently checked the most interesting is Helgen, which the FDA decided was good enough to stay on the market despite a relatively low positive predictive value – the probability that subjects with a positive test result truly have Covid-19 antibodies – of 68%.
| Independently evaluated Covid-19 antibody tests with EUAs | |||||
|---|---|---|---|---|---|
| Company | Test | Sensitivity | Specificity | PPV | NPV |
| Chembio Diagnostic | DPP Covid-19 IgM/IgG | 82.1% | 81.2% | 18.7% | 98.9% |
| Euroimmun (Perkinelmer) | Anti-Sars-CoV-2 Elisa IgG | 90.0% | 100% | 100% | 99.5% |
| Hangzhou Biotest Biotech | RightSign Covid-19 IgG/IgM | 100% | 100% | 100% | 100% |
| Healgen | Covid-19 IgG/IgM | 100% | 97.5% | 67.8% | 100% |
| Commercial tests only. PPV & NPV = positive & negative predictive values. PPV and NPV calculated at 5% prevalence. Source: FDA. | |||||
It is important to note that the tables above and below are just summaries – they do not include confidence intervals, for instance, or the sizes of the samples on which accuracy values were calculated. When a product tests separately for both IgG and IgM, combined accuracy figures are given. The FDA’s own page on antibody test performance has a fuller picture, including accuracy data for the separate antibodies where appropriate.
As for the 13 commercial tests that have received EUA but have not yet been independently validated, all look good enough to meet the FDA’s minimum standards, though the positive predictive value of Cellex’s test is markedly low at 55%. And no matter how decent these figures look, NCI scientists might come up with different results.
