Can real-world evidence studies replicate the results of randomized clinical trials? Insights from RCT-DUPLICATE

In the newly published RCT-DUPLICATE study, the design of 30 completed and 2 ongoing randomized controlled trials was emulated with real-world evidence studies using insurance claims data, reaching similar conclusions reported in the original studies where the designs were closely matched.

A collaboration between Brigham and Women’s Hospital (Harvard Medical School), Aetion and the US Food and Drug Administration (FDA), the RCT-DUPLICATE demonstration project is investigating whether real-world evidence (RWE) studies can emulate the findings from randomized controlled trials (RCTs) as a reference standard for valid causal inference in certain situations. New research, published online in JAMA, corroborates recent findings from the project and suggests that replicating RCT results using RWE studies is a possibility, highlighting the potential of RWE studies to complement RCTs in evaluating treatments and interventions in real-world settings, particularly when RCTs are not feasible or ethical. The study adds to the weight of evidence supporting the application of and trust in RWE to support regulatory decision making.

For their study, led by Shirley Wang and Sebastian Schneeweiss from Brigham and Women’s Hospital, the researchers used three US claims databases (Optum Clinformatics, MarketScan, and Medicare) to emulate the corresponding RCTs. RCTs were selected based on feasibility considerations such as statistical power, identification and control of key confounding factors, and the likelihood of being able to replicate the study endpoints using real-world data (RWD). In total, 32 studies were selected, 30 of which were completed and 2 ongoing at the time of the emulation. A transparent and reproducible process, including a common protocol template, was used to emulate each trial. All individual protocols were made available on clinicaltrials.gov. Each RWD database study was designed to match the design of its paired RCT as closely as possible, using similar parameters such as population, intervention, comparator and outcomes. The database emulations were conducted from 2017 through 2022 by Aetion, using their propriety Aetion Evidence Platform.

The researchers observed an overall positive correlation between the results of RCT-database pairs, as indicated by a higher Pearson coefficient (r=0.82). In a post-hoc analysis, half of the studies more closely emulated their RCT study pair in terms of design parameters, and a higher correlation in results was seen (r=0.93). In the other 16 studies, where close emulation with the original design was not possible, the correlation was lower (r=0.53). Lack of alignment of RCTs with real-life clinical practice, as well as other supporting factors, led to a weaker agreement in findings and persists as one of the main challenges for the emulation of RCTs using RWD. The authors explain, “Concordance in results varied depending on the agreement metric. Emulation differences, chance, and residual confounding can contribute to divergence in results and are difficult to disentangle.”

The authors conclude:

In post-hoc analyses, results of database studies were closely concordant with those of RCTs when the trial’s design and measurements could be closely emulated; however, database studies are not a substitute for RCTs. RCTs remain the standard for evidence generation on the efficacy of medical products for good reason. However, database studies can provide valuable complementary evidence by answering important questions on treatment effects in clinical practice that are not answered by RCTs.

---

You may also be interested in:

• Latest findings from the RCT-DUPLICATE demonstration project: an interview with Shirley Wang, Harvard Medical School
• How the FDA are using real-world evidence in regulatory decision-making: an interview with Nicholaas Honig, Aetion
• To replicate or not to replicate? Insights and interpretations from a randomized trial duplication initiative