By Deborah Borfitz

May 28, 2020 | Researchers are accustomed to having their ideas held up to scientific scrutiny, and testing and building on each other’s work, because that’s what moves the field toward more accurate explanations for problems and probable solutions. But the academic banter has largely devolved into discourses at the extremes during the current pandemic.

“Trying to find a middle ground has been excruciating,” says epidemiologist Marissa Carter, president of Bozeman, Montana-based Strategic Solutions, Inc., and staunch advocate of evidence-based medicine. Her company specializes in study design, protocol writing and study analysis, and pivoted almost entirely to COVID-19 treatment trials in March.

Seemingly everything about the novel coronavirus—including how fast a vaccine can be developed, if disputed drugs deserve presidential-level support and how COVID-19 cases and deaths get reported—has become a political flashpoint, says Carter. The consequences for science over the near and long term are potentially dire.

Carter is one of a growing number of health researchers and biomedical ethicists who have begun speaking out with the message that “there are no shortcuts” to defeating the virus. Circumventing basic principles of science and fairness will do more harm than good, adding to the death toll and enlarging inequities, they warn.

Rookie Mistakes

Jonathan Kimmelman, Ph.D., director of the biomedical ethics unit at McGill University in Montreal, is among those “horrified” by relaxation of research standards which, he notes, extend to scientific journals and newspapers of record. In a recent Science article, he and his longtime collaborator Alex John London (Carnegie Mellon University) argued “against pandemic research exceptionalism” and specifically flagged a New York Times article reporting the advantage of hydroxychloroquine in a study out of China.

“It took me literally 30 minutes, max, to find and read the preprint and look at the clinical trial registration to uncover all sorts of discrepancies between what the scientists said they were going to do and what they actually reported,” says Kimmelman. The reporter, the experts she consulted for the article, and perhaps also those maintaining preprint servers—now-massive repositories for scientific papers before they undergo peer review—made a “rookie mistake” by failing to ensure findings were presented in a balanced way.

Even for papers going into a preprint server, authors need to “come clean” about their methods, any deviations from the protocol and limitations of the study such as statistical powering or observational design features, he continues. The reporter was also culpable, by failing to check a public registry and critically engage her sources, as was the Times, since those quoted in the article were “favorably inclined toward the publication.”

Articles like this can create recruitment challenges for randomized clinical trials (RCTs), says Kimmelman, especially if patients believe they can get a promising drug from their regular doctor rather than enroll in a study and potentially get a placebo. Physicians are also going to be less likely to recruit their own patients into a placebo-controlled study if an unsubstantiated article alters their clinical judgment.

“There are any number of examples in the history of medicine where trials took a long time to complete in part because patients and physicians had shifted their opinion before a drug was shown to be safe and effective,” says Kimmelman. In some instances, the heralded drug turned out to be unsafe, ineffective or both when RCTs were eventually run.

Reasonable Risks

In general, strong regulatory approval standards are also needed to ensure the burdens and costs of medical uncertainty are more fairly distributed in society and drug companies underwrite the costs of establishing safety and efficacy of their products, Kimmelman says. Clinical trials have fairly exacting eligibility criteria, so the uncertainty is endured by patients who are relatively healthy. In the context of care, where there’s a mix of patients with both good and bad prognostic indicators, risks tend to land disproportionately on the disadvantaged.

The push for fast answers has meant a lot of the new vaccine platforms aren’t necessarily getting as thorough a vetting as they normally would and should, says Kimmelman. RNA vaccines have been rushed into clinical testing without any evidence that they prevent infection in animals, which isn’t how vaccine development normally happens. Compressing the usual, decades-long vaccine development process could be unnecessarily exposing human volunteers to unknown hazards.

Human infection trials are another matter, Kimmelman says. In fact, he has helped develop an ethical framework for such studies, where a small number of volunteers are deliberately exposed to a pathogen to study infection and gather preliminary efficacy data on experimental vaccines or treatments. In their published statement, collaborating ethicists say they agree human infection trials can in principle have social value—and none of their disagreements are insurmountable.

The Miscounts

Carter has experience in infectious disease research that goes back more than a decade, so when COVID-19 hit she had clients asking her to prognosticate on the likely case count. Putting on her modeling hat, she crunched the numbers and came up with an infection rate model on March 1 showing the U.S. would hit the one million mark by early April.

Her model turned out to be far more accurate than other models, including one produced by Harvard that was only counting the confirmed cases. Carter was taking into consideration the testing backlog, insufficient testing, and asymptomatic cases. She shared her findings on the blog site of a physician friend.

But as new coronavirus cases continued to mount in New York City and California, Carter says, she was shocked like everyone else by the magnitude of the public health crisis. It was far worse than even she had imagined. Even by crude estimates, cases were being undercounted by a factor of at least 100%.

Sadly, she continues, some epidemiologists preferred pouncing on one another in lieu of fair-minded scientific debate. Survey sample sizes haven’t been big enough to confidently draw precise conclusions, but the most salient point is that COVID-19 prevalence is a “lot higher” than originally thought.

The cost effectiveness of our collective response to COVID-19 will be examined in the decades to come, says Carter, who has been monitoring the virus since last November when it was still just a curiosity in China. Half of all epidemics originate in China and, when they emerge, tend to be “covered up badly.”

Interpolating the data is the best a researcher can do. In a modelling study published earlier this year in The Lancet, mathematical modelling was needed to estimate the size of the epidemic in China, and its potential future spread domestically and internationally, by combining officially recorded case data with domestic and international travel data and some educated guesses (e.g., the time it would take for infected individuals to infect other people would mirror that of SARS).

Estimates were that the epidemic was doubling every 6.4 days, and that other countries would likely be at risk of experiencing epidemics during the first half of 2020. Around the same time, researchers in the U.K. were also sounding the alarm about the undercounting of cases and focus on a crudely calculated 2% case fatality rate based purely on symptomatic cases, says Carter.

Given the dearth of information about the virus, only two modelling tools even remotely make sense, says Carter, including the “moving average with tweaks” approach that often works well for short-term projections and social models built around the movement and activities of people. But neither can provide an accurate case count for COVID-19 in the U.S., both because the testing program in the first couple of months of the pandemic was “absolutely appalling” and cases have been counted based on clinical symptoms that may be mild to non-existent.

That the pandemic has silent spreaders, with unknown immunity status, has been extremely challenging even for highly experienced epidemiologists, says Carter. “It’s like being back in grade school.”

The politics of COVID-19 aside, it is difficult to know when the disease is the last straw that kills a person when there are other comorbidities and problems that were contributing factors, Carter says. But whether states choose to uniformly attribute all deaths to COVID-19, even without a confirmed positive test, or to not count COVID-19 cases at all when the diagnosis isn’t believed to be a contributing factor, science is being muddied in “unprecedented” ways.

“A lot of epidemiologists are scratching their heads,” she says, because to understand disease transmission and protect people requires collaboration across state and party lines. Instead, the U.S. is beset by divided sentiments about the status of the COVID-19 situation, miscommunication to public health officials, and a rising number of conspiracy theories that are running rampant on social networks.

Admittedly, Carter herself held a decidedly different epidemiological point of view only a few months ago when it came to the topic of social distancing. “How I feel about people who are breaking lockdown has changed. There has to be a balance. The need for mental health in this country a year from now is going to be staggering” and, she adds, minority and underprivileged populations hardest hit by the pandemic are also the ones who can least afford such services.

History Repeats Itself

Traditionally, results of clinical trials and epidemiological work didn’t get discussed until they were peer-reviewed and published in a scientific journal. But the pandemic has created an explosion of pre-prints still in the queue that sometimes make the headlines in mainstream media outlets, says Carter.

A few months back, a team of Indian scientists had a preprint version of their research paper posted that found HIV sequences in the coronavirus genome, and the news rocked the social media news, she cites as an example. Other scientists immediately “jumped down their throats” in an online free-for-all without any proper refereeing. A more appropriate response might have been to do a thorough review of the study and thoughtfully comment on the discovery or point out that it is just an artifact of no consequence.

High-level excitement about the promise of hydroxychloroquine—no doubt prompted by a few lucky Hail Mary passes by desperate clinicians on the pandemic’s front lines—was perhaps the bigger fumble because it completely ignored lessons learned in the aftermath of the 2002-2004 severe acute respiratory syndrome outbreak, says Carter.

In recent weeks, two large and well-controlled observational trials have been published indicating hydroxychloroquine is “not only not helpful but it makes things worse,” notes Carter. But the White House also confirmed that President Trump has been taking the drug, with his doctor’s blessing, which has reignited controversy about its value.

“Medical research is always about testing medical hypothesis and assumptions and when you’re at the beginning of something new like this you are more likely to be wrong than to be right,” says Carter, drawing a parallel with scientific thought on HIV infection during most of the 1980s. “It’s almost like we had to reinvent ourselves to understand what that bloody virus was doing to everybody”—a scenario she expects will be happening with COVID-19 only at a more accelerated pace.

“It’s sad to say, but history is repeating itself again because we failed to learn what we should have learned 30 years ago,” she says. And it will take many years to retrospectively calculate the damages.

As reported in BMJ, the lesson learned in the aftermath of the three earlier SARS outbreaks in China included honesty is needed, controversy leads to lost chances and that conclusions may be premature. The disease was initially called “atypical pneumonia” and health authorities said it was caused by Chlamydia and could be treated with antibiotics, prompting some research centers to carry out trials to support those findings.

The rush for answers isn’t limited to drugs. When COVID-19 cases in the U.S. first started piling up in hospitals, the prevailing assumption was that it was going to be all about intensive care and ventilators because of the connection between coronaviruses and acute respiratory distress syndrome (ARDS). But within weeks it became obvious that putting someone with ARDS on a ventilator is “the kiss of death,” says Carter. “COVID-19 gets into the bloodstream quite easily and causes all kinds of multiple organ failures and attacks on the vasculature.”

Because of the fixation on producing ventilators, “we missed the boat on other things,” she continues. “If you look at a snapshot of how a patient got treated on March 1 versus May 1, you’re going to see two quite different ways of dealing with the same problem.”

Accelerating the learning curve, Carter says, requires constantly asking, “Are we doing the right stuff?”

Emergency Use Authorizations

The Food and Drug Administration (FDA), an administrative entity whose functioning depends on the support of congress and the executive branch, has been using its Emergency Use Authorization (EUA) authority to expedite the study of therapeutics for COVID-19. The approach raises a few ethical red flags, says Kimmelman, including the fact that many people wrongly infer a product with EAU status has been deemed effective by the agency.

This “signaling” of a product’s effectiveness may potentially interfere with proper evaluation of the intervention, opening up the possibility that many people will be exposed to an unsafe and ineffective treatment, Kimmelman says. Given that healthcare budgets are extremely tight right now, with many small hospitals closing and many families struggling just to feed their family, it’s questionable if buying large quantities of an unproven intervention is the best use of taxpayer dollars, he adds.

The FDA, as well as people sitting on data safety monitoring boards, started getting “terribly worried about the safety of patients” by early March, says Carter. The agency, to its credit, quickly responded with guidance for industry that “forced almost everyone in the business to rethink what they were doing,” she says.

The volume of new trial starts, coupled with an ongoing staff shortage, also forced the FDA to “completely restructure” its trial approval process, add Carter. That is likely helping to rein in any “cowboys” trying to make false or misleading claims about drugs under study.

Flawed Process

The scholarly peer review process is itself a flawed, but necessary, process that has been strained in new ways by the pandemic, says Carter. Early in 2020, many of the leading medical journals purposefully expedited the production process for COVID-19 articles from months to weeks and made them all publicly accessible.

“But you can’t do that for every paper, which means you have to start selecting things via an editorial board with a certain bias,” she continues. They could be “totally wrong” about which studies to feature. The issue for high-profile journals is partly the rejection rate on the deluge of submissions and, at the lower-tier journals, less experienced reviewers. Peer reviewers faced with a blizzard of papers can also get fatigued to the point of making bad decisions.

As reported in the New York Times in mid-April, preprint servers and peer-reviewed journals are seeing surging audiences and not all of them understand the limitations of the latest research findings. COVID-19 scientific news, whether formally evaluated or not, is making the headlines and some of it has become fodder for conspiracy theorists.

Scientists, for their part, are personally responsible to “be transparent about weaknesses in their studies,” says Kimmelman. This is a common shortcoming in the preprint literature, he adds.

The scientific community was “caught unprepared” for such a heavy reliance on preprint publications in terms of their speed and breadth of circulation, Kimmelman says. It might have been useful to assemble peer-review “shock troops” so incoming papers would get immediate, rigorous critique—and journalists and other readers would be alerted to any of the obvious shortcomings.

“It is certainly not hard to find examples of reports and studies with deficiencies that under normal circumstances and settings would be recognized as unacceptable but … people consider acceptable during this pandemic,” says Kimmelman.

COVID-19 aside, the economics of research are such that in a market where sloppy science is neither recognized nor punished—and perhaps even celebrated—upstanding scientists will have a much tougher time competing for funding, attention, and promotions, says Kimmelman. The flashy, headline-grabbing research will win out every time. “This is a general challenge we have in drug development.”

Master Protocols

A number of research initiatives are underway “trying to right the ship” and reward conscientious science, Kimmelman says. The World Health Organization (WHO), for example, is running a well-designed, multi-country trial testing four potential COVID-19 treatments head to head with a single placebo arm. The study is using a master protocol, a type of adaptive trial design that involves adding and removing drugs, arms, and hypotheses from an ongoing trial.

The initial treatment arms are remdesivir; the combination of two HIV treatments, lopinavir and ritonavir; lopinavir and ritonavir plus interferon beta; and chloroquine. The trial will be augmented by data collected in a separate European study using a master protocol that has also been developed by WHO.

When compared to traditional trials for each of the four drugs, WHO’s study requires three fewer placebo study arms, is less expensive to conduct and provides more useful information, says Kimmelman. Statistical measures are being taken to mitigate the risk of false-positives from testing multiple hypotheses with a single set of data.

Master protocols are too financially risky for most drug companies to embrace, says Kimmelman. Since companies are rewarded for demonstrating product performance against a placebo rather than a competitor’s drug, they have more wiggle room for marketing its virtues—and, potentially, a whole lot less to lose than in a direct competition.

The fact that almost all of the drugs being tested against COVID-19 are already sitting on the shelf is making it easier to run master protocols, says Kimmelman. “You don’t necessarily have to have signoff from the drug company to do these trials. Federal funding agencies, governments and consortia can orchestrate their own master protocol.”

The global nature of the WHO trial will ensure that the findings are broadly applicable to everyone, adds Carter, which is great. On the other hand, responses to treatment can vary considerably from one country to the next so it may turn out that more region-specific trials are also needed. What works well in Italy, with its high proportion of elderly residents, may do better or worse in countries like Taiwan and New Zealand with a younger demographic profile.