The Question of Mitigating Patient Mortality: Comparing Gilead’s ACTT-1 and the WHO’s Solidarity Trials
By Peter Pitts
Science is hard. The last time many adults thought about science was in grade school when there was a right and wrong answer. In reality, science is complex and expensive, the best work mostly results in failure, and success is relative. But it’s that “relative success” that matters most. As any medical scientist will tell you, there are few “Eureka!” moments in health research. “Miracle cures” are for the movies. Incremental innovation is the rule—and it can change the world.
The world has learned that, when it comes to COVID-19, there are no easy answers, no quick fixes, no magic bullet. The development of diagnostics, therapeutics, and vaccines are taking place at “warp speed.” An important result of that acceleration has been a rethinking of the product development status quo, including new ways to make better ventilators and new techniques for vaccine development and manufacturing (such as mRNA technology that is more precise, safer, and less expensive to produce). Other key status-quo-busters include new ways to design, field, and analyze data from clinical trials while ensuring sufficient nimbleness in the FDA review process to expedite robust regulatory decision-making that can save lives.
Consider remdesivir (Veklury), Gilead’s FDA-approved therapy for patients with serious manifestations of COVID-19. Originally made available to patients under an Emergency Use Authorization (an Emergency Use Authorization—EUA—allows an experimental product to be used more broadly than just for clinical trials), remdesivir is currently the only new therapeutic approved by FDA for the COVID-19 armamentarium. FDA’s decision to issue the full approval was based on solid evidence.
In a large double-blind, randomized, placebo-controlled trial named the Adaptive COVID-19 Treatment Trial (ACTT-1) and led by researchers at the National Institute of Allergy and Infectious Diseases (NIAID), 1,062 hospitalized subjects with mild, moderate, and severe COVID-19 received either remdesivir or a placebo, plus standard of care. The median time to recovery for the remdesivir group was ten days, compared to fifteen days for the placebo group. Overall, the odds of clinical improvement at Day 15 were significantly higher in the remdesivir group when compared to the placebo group. Final trial results were peer-reviewed and reported in the New England Journal of Medicine (NEJM) in early October of this year. Data demonstrated that remdesivir helped hospitalized COVID-19 patients to recover faster compared to placebo. Remdesivir isn’t a cure, but it can be used to save lives that would be lost without it.
Science isn’t static. On October 15th, the World Health Organization (WHO) researchers, working on a study named Solidarity, released a preliminary report showing that nearly 3,000 people receiving remdesivir were not more likely to survive infection with SARS-CoV-2 than those receiving only standard of care. The WHO preliminary findings seem to be in direct contrast to the findings of the Adaptive COVID-19 Treatment Trial 1 (ACTT-1). Who should we believe? What do the conflicting results mean?
Science is a series of gray zones and interpretation of data in context is key. The first thing to remember is that the WHO Solidarity study is not a replication of ACTT-1, and that is a distinction with a difference. Let’s compare and contrast.
First, the WHO’s Solidarity study was conducted in dozens of countries with a variety of different approaches to standard of care, and the cross-national data are difficult to analyze and compare. Next, the Solidarity study was not double (or even single) blinded. Such so-called “open-label” trials can often skew the reporting of results because patients and their doctors know what treatment the patient is on. Further, the Solidarity study reports primarily on the general population and provides limited results in subgroups. The ACTT-1 data (as reported in the NEJM article) make it clear that remdesivir has not shown a statistically significant mortality improvement in the general population but is effective in patients during low-flow oxygen therapy and the medium-to-severe phase of COVID-19. In other words, remdesivir can help to save the lives of those most at-risk, especially when used early in the disease course.
The many differences in the way the WHO and NIAID studies were conducted make it difficult to reconcile the results. The WHO study, for instance, only counted deaths that occurred in the hospital but not those that might have occurred in patients who were discharged. Solidarity does not clarify the time of treatment. This is important because remdesivir is very effective when used in the first ten days of infection. The Solidarity study also had no data monitoring, no placebo, no double-blinding, inconsistent diagnostic confirmation of infection, no reported timing of symptom duration before treatment initiation, unknown baseline physiological severity, unknown supportive care provided, unknown health care capacity status of enrolling sites, and a large amount of missing data. According to Dr. Andre Kalil, a professor of internal medicine at the University of Nebraska (who was among the first to treat patients coming off the Diamond Princess cruise ship with remdesivir), “Poor quality study design cannot be fixed by a large sample size, no matter how large it is.”
Even if we were to accept Solidarity’s reported findings at face value, what does it actually tell us about mortality?
If you dig into WHO’s data on mortality, you find an interesting and important outcome. WHO’s preprint manuscript contained a meta-analysis of existing studies of remdesivir. The preprint reported a 99% confidence interval, rather than a 95% confidence interval, in the subtotal of the largest population: non-ventilated patients. This is unusual, given that a similar meta-analysis on steroids for COVID-19 used a 95% confidence interval across all studies and for subtotals. That meta-analysis was authored by the WHO Rapid Evidence Appraisal for COVID-19 Therapies (REACT) Working Group.
If we apply the same meta-analytic approach for remdesivir that WHO used for steroids, an important finding becomes immediately clear. Across all studies, among COVID-19 patients not on ventilators, there is a statistically significant 20% (RR=0.80, 95% CI: 0.67, 0.95) reduction in mortality with remdesivir treatment compared to control. Note that the vast majority of patients (3,309 of 3,818, or 87%) receiving remdesivir (Table 1) were not ventilated. Keeping a fifth of those people alive (despite the limitations of the study) seems like a pretty big deal. One would think that WHO would want to shout this from the medieval Geneva rooftops.
Coincidentally, the Institute for Clinical and Economic Review (ICER) updated their report on remdesivir’s value on November 10. ICER removed its assumption of a mortality benefit for remdesivir on the grounds of WHO’s meta-analysis. Would they have made a different choice if WHO had presented the data consistently with its other meta-analysis? No one knows, but let’s hope that WHO and ICER’s actions do not lead to access restrictions that hurt patients.
In the influential NEJM, an editorial by David Harrington at the Harvard T.H. Chan School of Public Health, infectious disease specialist Dr. Lindsey Baden, and Brown University biostatistician Joseph Hogan cites problems with Solidarity. They write that the study does not refute trials that demonstrated benefits of remdesivir in treating COVID-19. They pointed out that the thirty-nation Solidarity study had inconsistencies in the data collected due to “variation within and between countries in the standard of care and in the burden of disease in patients who arrive at hospitals.”
According to Daniel Rubin, Kirk Chan‑Tack, John Farley, and Adam Sherwat of, respectively, the FDA’s Division of Biometrics IV, Office of Biostatistics, the Division of Antivirals, Office of Infectious Diseases, and the Office of Infectious Diseases, the findings of the Solidarity study are “not inconsistent with ACTT-1, which also did not establish a mortality difference between remdesivir and placebo. However, ACTT-1 did demonstrate robust results on time to recovery and odds of clinical improvement in a placebo-controlled, double-blind trial, whereas Solidarity was not designed to rigorously assess these end points.”
More science is better. While both the Solidarity and ACTT-1 studies contribute to our understanding of interventions to help treat COVID-19, the two clinical trials had different trial designs and primary goals. The design of ACTT-1 (i.e., randomized, placebo-controlled, double-blinded) was better suited to rigorously assess a time to recovery endpoint compared to a trial with an open-label design, such as the Solidarity trial. FDA agrees that, based on the findings of the ACTT-1 trial, the benefits of remdesivir include a shorter time to recovery and better odds of clinical improvement.
But the mortality question is important as well. The headlines on Solidarity do not match the data. Applying a 95% confidence interval suggests a critical mortality benefit that could help advance our fight against COVID-19. Let’s hope that thoughtful stakaeholders hold WHO accountable and ensure that we get an evidence-based understanding of remdesivir and other potential COVID-19 treatments and vaccines.
 https://www.hhs.gov/coronavirus/explaining-operation-warp-speed/index.html Last accessed November 11, 2020.
 https://www.nasa.gov/feature/jpl/nasa-develops-covid-19-prototype-ventilator-in-37-days Last accessed November 11, 2020.
 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3597572/ Last accessed November 11, 2020.
 https://www.tandfonline.com/doi/abs/10.1080/15265161.2020.1795516 Last accessed November 11, 2020.
 https://www.statnews.com/2019/10/14/fda-smashing-status-quo-regulatory-science/ Last accessed November 11, 2020.
 https://www.vekluryhcp.com/?gclid=Cj0KCQiA7qP9BRCLARIsABDaZzjIhV7OdumdHsh4tYdydeUOHemZv4fc-HOYpuAV2nJxyY5KN4MOsK0aAmTuEALw_wcB&gclsrc=aw.ds Last accessed November 11, 2020.
 https://www.fda.gov/media/137564/download Last accessed November 11, 2020.
 https://www.fda.gov/news-events/press-announcements/fda-approves-first-treatment-covid-19 Last accessed November 11, 2020.
 https://www.nejm.org/doi/full/10.1056/NEJMoa2007764 Last accessed November 11, 2020.
 https://www.medrxiv.org/content/10.1101/2020.10.15.20209817v1.full.pdf Last accessed November 11, 2020.
 https://time.com/5901087/remdesivir-conflicting-studies/ Last accessed November 11, 2020.
 https://www.bmj.com/content/370/bmj.m3379 Last accessed November 11, 2020.
 https://icer-review.org/announcements/ Last accessed November 11, 2020.
 https://www.nejm.org/doi/full/10.1056/NEJMe2034294 Last accessed December 3, 2020.
 https://www.nejm.org/doi/pdf/10.1056/NEJMp2032369 Last accessed December 3, 2020.