A critical examination of the controversial study behind hydroxychloroquine and azithromycin for COVID-19

On March 20th, the International Journal of Antimicrobial Agents published Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial by Gautret et al.  The president of the United States tweeted about the article the very next day, and on March 28th the FDA announced an emergency use authorization to allow the distribution of hydroxychloroquine and chloroquine from national stockpiles to treat patients hospitalized with COVID-19.

Since then, this study has come under intense scrutiny.  On Friday the International Society of Antimicrobial Therapy released a statement of concern about the paper it’s journal had just published, writing that “[t]he ISAC Board believes the article does not meet the Society’s expected standard,” and “[while] it is important to help the scientific community by publishing new data fast, this cannot be at the cost of reducing scientific scrutiny and best practices.”

So what was the study, and what are the concerns?

First, some background.  Chloroquine and hydroxychloroquine are antimalarial and antinflammatory drugs with in vitro activity against a number of viruses, including SARS-CoV-2 (hereafter, COVID-19).  However, drugs that kill viruses in laboratory cell cultures frequently fail when used in patients.  Chloroquine inhibits influenza in vitro, yet did not prevent influenza infection in a clinical trial; similarly, chloroquine inhibits the replication of chikungunya in vitro, but had no effect when prescribed to patients during a 2006 outbreak, and actually enhanced chikungunya infection in a primate model.

Gautret et al reported the viral PCR testing outcomes of 36 patients with COVID-19 in France, of whom 6 received hydroxychloroquine and azithromycin, 14 received hydroxychloroquine alone, and 16 received neither agent.  The authors report that 100% of patients who received both hydroxychloroquine and azithromycin had a negative PCR by day six of the study, versus only 57% of the patients who received hydroxychloroquine and only 12.5% of the controls. They conclude that “hydroxychloroquine treatment is significantly associated with viral load reduction/disappearance in COVID-19 patients and its effect is reinforced by azithromycin.”

Let’s set aside the fact that we don’t know whether changes in viral load have anything to do with how patients fare clinically.  This sounds great, right?

Unfortunately, this study had problems. Academic watchdog and author of Science Integrity Digest Dr. Elizabeth Bik described several of the issues with this paper at length, but to summarize the key points:

1. Six patients originally assigned to the drug treatment arms were described as “lost to followup” because their treatments were stopped early: of these, one died and three were moved to the ICU. Of course, any drug will look great for COVID-19 if you don’t count the patients who received it and then got worse or died.

2. Critical data is missing in a way that skews the reported outcomes. For example, the authors report that only 12.5% (2/16) of the control patients had negative viral PCRs at day 6, but what is left unsaid is that a third of the control patients didn’t have PCR testing on day 6, so there was no way to know if they were negative.

Further, the PCR data is reported strangely.  First, some tests are reported with specific CT value (cycle threshold; the number of times you need to duplicate the RNA in a sample to generate a detectable amount), whereas others are reported as positive or negative. Are the authors using multiple different PCR tests and reporting them together?  Second, CT values can vary widely between PCR tests, and even from batch to batch when using the same test platform – meaning that because the authors did not run each patient sample along side a standard curve of known quantities of SARS-CoV-2 RNA, the observed changes in CT values may not have any relation to changes in the patients’ viral loads (i.e. may be totally meaningless)

3. The study’s methods imply that the methodology was changed after the trial began. Note that the methods indicate that patients were to be treated for 10 days and that the power calculation for the study was based on reduction in the viral load at day 7. Yet the primary outcome is given as “viral clearance” at day 6. Moreover, the original study protocol submitted to the EU clinical trials registry stated that PCRs would be collected on days 1, 4, 7, and 14.  Why the change?  Note also that while the study states multiple clinical parameters would be secondary outcomes (e.g normalization of temperature and respiratory rate or length of hospital stay), none of these are reported in the results.  The authors state that this is a preliminary report of an ongoing study, published early because their PCR findings were just so important – but isn’t how the patients fared clinically what really matters?

As an aside, this clinical trial was approved by the French ethics board on March 6th and submitted for publication on March 16th while reporting a total 7 days of data.  Were the authors able to both enroll all of their patients and write their manuscript in less than 72 hours, or were they were recruiting patients into a clinical study before getting ethics approval?

4. Patients in the drug treatment and control arms were recruited from different medical centers (recruitment into the treatment arm occurred only in Marseille), and at the Marseille site patients who either refused treatment or who had one of the study’s exclusion criteria were recruited as controls.  So, not only was the study non-randomized, the introduction of confounding factors was baked into the study’s design.

5. The paper was submitted for consideration on March 16th and accepted March 17th.  For this paper to have passed peer review in less than 24 hours (indicating the reviewers had no substantive criticisms, which is not surprising given that the version published is essentially identical to the pre-print release) is truly remarkable, particularly given that the IJAA’s editor in chief is listed as a co-author on the paper.

Physicians and patients around the world are understandably desperate for treatments for COVID-19 infection.  In my view, the study by Gautret et al is critically flawed and does not support hydroxychloroquine and azithromycin as an effective treatment for COVID-19 infection. Unfortunately, this study has created a fervor around hydroxychloroquine and azithromycin that has led to national shortages and anecdotal reports of patients with lupus and other autoimmune diseases (which hydroxychloroquine actually treat) being unable to fill their prescriptions. One person has already died after self-medicating with a chloroquine solution sold for aquariums in an attempt to prevent COVID-19 infection. How many deaths from wanton use of these two agents – both of which prolong the QT interval, promoting cardiac arrhythmias – will go unreported?

In a recent JAMA editorial, our UNMC ID colleague Dr. Andre Kalil put it best, arguing that the use of unproven drugs for COVID-19 outside of the context of randomized control trials not only risks harm to patients but delays the work of discovering which therapeutics actually work. He writes, “[a]lthough many drugs have in vitro activity against different coronaviruses, no clinical evidence currently supports the efficacy and safety of any drug against any coronavirus in humans, including SARS-CoV-2. Numerous drugs that have been highly promising in vitro for other infectious diseases have failed in clinical studies…. A common interpretation of off-label use and compassionate use of drugs is that is that if the patient died, they died from the disease, but if the patient survived, they survived because of the given drug. This is not true.”  Studies like the trial by Gautret et al do not advance the science, and in kindling enthusiasm for drugs with potentially fatal side-effects, may well do harm.