COVID-19 vaccine trials cannot tell us if they will save lives

Vaccines are being hailed as the solution to the COVID-19 pandemic, but the vaccine trials currently underway are not designed to tell us if they will save lives, reports Peter Doshi, Associate Editor at The BMJ today.

Several COVID-19 vaccine trials are now in their most advanced (phase 3) stage, but what will it mean exactly when a vaccine is declared "effective"?

Many may assume that successful phase 3 studies will mean we have a proven way of keeping people from getting very sick and dying from COVID-19. And a robust way to interrupt viral transmission.

Yet the current phase 3 trials are not actually set up to prove either, says Doshi.

"None of the trials currently underway are designed to detect a reduction in any serious outcome such as hospitalisations, intensive care use, or deaths. Nor are the vaccines being studied to determine whether they can interrupt transmission of the virus," he writes.

He explains that all ongoing phase 3 trials for which details have been released are evaluating mild, not severe, disease—and they will be able to report final results once around 150 participants develop symptoms.

In Pfizer and Moderna's trials, for example, individuals with only a cough and positive lab test would bring those trials one event closer to their completion.

Yet Doshi argues that vaccine manufacturers have done little to dispel the notion that severe COVID-19 was what was being assessed.

Moderna, for example, called hospitalisations a "key secondary endpoint" in statements to the media. But Tal Zaks, Chief Medical Officer at Moderna, told The BMJ that their trial lacks adequate statistical power to assess that endpoint.

Part of the reason may be numbers, says Doshi. Because most people with symptomatic COVID-19 infections experience only mild symptoms, even trials involving 30,000 or more patients would turn up relatively few cases of severe disease.

"Hospitalisations and deaths from COVID-19 are simply too uncommon in the population being studied for an effective vaccine to demonstrate statistically significant differences in a trial of 30,000 people," he adds. "The same is true regarding whether it can save lives or prevent transmission: the trials are not designed to find out."

Zaks confirms that Moderna's trial will not demonstrate prevention of hospitalisation because the size and duration of the trial would need to be vastly increased to collect the necessary data. "Neither of these I think are acceptable in the current public need for knowing expeditiously that a vaccine works," he told The BMJ.

Moderna's trial is designed to find out if the vaccine can prevent COVID-19 disease, says Zaks. Like Pfizer and Johnson and Johnson, Moderna has designed its study to detect a relative risk reduction of at least 30% in participants developing lab-confirmed COVID-19, consistent with FDA and international guidance.

Zaks also points to influenza vaccines, saying they protect against severe disease better than mild disease. "To Moderna, it's the same for COVID-19: if their vaccine is shown to reduce symptomatic COVID-19, they will feel confident it also protects against serious outcomes," Doshi writes.

But Doshi raises another important issue—that few or perhaps none of the current vaccine trials appear to be designed to find out whether there is a benefit in the elderly, despite their obvious vulnerability to COVID-19.

If the frail elderly are not enrolled into vaccine trials in sufficient numbers to determine whether there is a reduction in cases in this population, "there can be little basis for assuming any benefit against hospitalisation or mortality," he warns.

Doshi says that we still have time to advocate for changes to ensure the ongoing trials address the questions that most need answering.

For example, why children, immunocompromised people, and pregnant women have largely been excluded; whether the right primary endpoint has been chosen; whether safety is being adequately evaluated; and whether gaps in our understanding of how our immune system responds to COVID-19 are being addressed.

"The COVID-19 vaccine trials may not have been designed with our input, but it is not too late to have our say and adjust their course. With stakes this high, we need all eyes on deck," he argues.

• by British Medical Journal
• SARS-CoV-2 (shown here in an electron microscopy image). Credit: National Institute of Allergy and Infectious Diseases, NIH