By Allison Proffitt

September 6, 2018 | Major healthcare and reimbursement decisions are made based on causal evidence for treatments and interventions, and it helps if that evidence to be accurate.

“Real-world evidence is about utilizing the electronic data that are generated by healthcare systems through insurance claims, through electronic health records… to understand how medical interventions, and medical products like medications and devices work in routine care,” Sebastian Schneeweiss, Professor of Medicine at Harvard Medical School, told Clinical Informatics News.

These data are of great use to payers, providers, and regulatory bodies as they decide which treatments to prioritize, which to pay for, and how to indicate secondary indications.

“What the field was struggling with for many years was a simple intransparency and inability to replicate studies that people have done,” Schneeweiss said. “They used these very large databases, run analyses, and then other people cannot reproduce or understand what was done. That’s a major problem! How can you trust the results if you don’t understand and cannot even replicate what people have done?”

It was a problem Schneeweiss has been tackling for 20 years at Harvard Medical School, and it’s the goal of the REPEAT initiative, a non-profit program from Brigham and Women’s Hospital (BWH) and Harvard Medical School (HMS) committed to improving the transparency, reproducibility and validity of longitudinal healthcare database research.

Schneeweiss; Shirley Wang, director of the REPEAT program and assistant professor of medicine at Harvard Medical School; and several other researchers and students are working to improve the reproducibility of database research.

“We’re trying to improve the transparency with which we conduct database research to try to help decision-makers, regulators, and healthcare organizations to be able to better interpret and assess the validity and the relevance of the insight that is generated from databases,” Wang says.

REPEAT started with a large, random sample of published studies that were conducted using publicly-available data sources. The team is evaluating the transparency of reporting a list of specific parameters that are often hidden in the methods or appendices of a publication. Then, one-by-one, the research team is replicating the studies.

“We’re basically going through this catalogue and seeing, Can we tell what they did for the decision? Or do we need to make an assumption when we try to replicate?” Wang explains. “We’re trying to understand what factors may enable us to better replicate or hinder us from being able to replicate.”

Platform Solution

It’s a big challenge, and to facilitate the process, the REPEAT program chose the Aetion Evidence Platform, a tool spun out of Schneeweiss’s Harvard lab about five years ago.

The Aetion Evidence Platform can analyze virtually any source of real-world data to determine medical need, comparative effectiveness, and the overall value profile of medical treatments and interventions.

“When you do these database studies, looking for relationships between drugs and outcomes, you realize that the process is actually quite repetitive from study to study,” Schneeweiss explained. “There are different parameter settings, and the study, of course, is always different, but the principle of how you do these studies is quite repetitive.”

Schneeweiss brought Jeremy Rassen on board, a computer scientist with a “Silicon Valley background” to program macros and tools to automate the process. “But then five-years ago, we decided in order to build really industry-strength platform, we needed to have very serious software engineers involved and that is just not doable in an academic setting,” Schneeweiss explained. “That was the point that we spun it off and started Aetion.”

Rassen now serves as co-founder, president, and chief science officer at Aetion; Schneeweiss is co-founder and science lead.

“When we do these types of [database replication] studies, it usually takes us 6, 9, or 12 months to do a single study. Clearly when we set out with this, we knew that in order to replicate 150 studies… we needed a platform like the Aetion platform,” explains Schneeweiss. “You can implement these studies at scale, at the same time, highly transparently, on a validated platform, with an audit trail, which line programing doesn’t allow.”

With the Aetion platform, and if the study has a well-defined “recipe”, the team can replicate a study in about a month. “We’re averaging about four weeks per study,” Wang says. “We’ve got probably 14 in progress.”

“This is the foundational work that our community needs, to understand the reproducibility of our work in general,” Schneeweiss said. “The platform is used to do studies, inform approval decisions, to predict findings from ongoing phase 4 trials.”

In addition to the REPEAT project, the Aetion platform is also being used in a large-scale FDA-sponsored study to replicate the results of 30 published randomized controlled trials. Coupled with REPEAT, the FDA study aims to validate whether the use of real-world evidence leads to the same regulatory decisions as randomized controlled trials and to demonstrate whether this evidence could be used to supplement or, in certain circumstances, even replace clinical trials for drug development and regulatory approval.

Early REPEATable Results

So far Wang’s team has reviewed transparency for 112 studies, with a goal of 250. They have replicated 50, with a goal of 150. Wang presented the program’s first interim results last month at the International Conference on Pharmacoepidemiology & Therapeutic Risk Management.

“The biggest issue is temporality of how people are defining things,” Wang says. “Even if they provide the codes for how they measure things, when they measured it relative to study entry, and the order of application of exclusions to create the study population [matters].”

The order with which exclusions apply redefines the study population, which of course impacts results. “If you can’t create the study population, you’re not going to get the same analysis results,” Wang explains. “These basic steps are often hand-waved, whereas the actual analytics methods are described in more detail.”

The program is now evaluating the robustness of evidence found in health care database studies and checking for appropriate research design and analysis. Final results are anticipated in late 2019.

“It turns out that this is turning into a global movement,” Schneeweiss says. “Partners in Taiwan and South Korea and Denmark and other jurisdictions are quite interested in doing similar things with their local data, which we don’t have available. What Shirley started is now turning more and more into a global movement, which is quite exciting.”