A recent scoping review published in The Lancet Digital Health examines the state of randomized controlled trials (RCTs) evaluating artificial intelligence in clinical practice, with a focus on medical imaging. The review aims to assess the effectiveness and generalisability of AI models integrated into clinical workflows.
The researchers conducted a comprehensive search of studies published between January 2018 and November 2023, focusing on trials in which AI was a significant component of patient management. In total, they analysed 86 RCTs and found that 81% reported positive outcomes, particularly in terms of diagnostic performance.
Authors suggest that to better assess the true value of AI algorithms in health care, it is crucial for real-world evidence to focus on clinically meaningful endpoints such as symptoms and need for treatment, as well as longer-term outcomes such as survival instead of merely diagnostic performance.
However, the review highlights concerns about the generalisability of these results due to the predominance of single-centre trials and limited demographic reporting. It highlights the need for more multi-centre trials and diverse outcome measures to fully understand the impact of AI in real-world settings.
Read full study
Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review
The Lancet Digital Health, 2024
Abstract
This scoping review of randomised controlled trials on artificial intelligence (AI) in clinical practice reveals an expanding interest in AI across clinical specialties and locations. The USA and China are leading in the number of trials, with a focus on deep learning systems for medical imaging, particularly in gastroenterology and radiology. A majority of trials (70 [81%] of 86) report positive primary endpoints, primarily related to diagnostic yield or performance; however, the predominance of single-centre trials, little demographic reporting, and varying reports of operational efficiency raise concerns about the generalisability and practicality of these results. Despite the promising outcomes, considering the likelihood of publication bias and the need for more comprehensive research including multicentre trials, diverse outcome measures, and improved reporting standards is crucial. Future AI trials should prioritise patient-relevant outcomes to fully understand AI's true effects and limitations in health care.