A prospective, single-center study in the United States evaluated Aidoc's AI tool for detecting intracranial hemorrhage (ICH) in noncontrast head CT (NCCT) examinations, involving 9,954 CT scans from 7,371 patients. The AI tool was implemented to triage and notify radiologists of ICH-positive cases via a pop-up display. The study included two phases: phase 1 (radiologists working without AI) and phase 2 (radiologists assisted by AI). The AI processed head NCCT scans, notifying radiologists of ICH-positive cases, but did not prioritize worklists.
The study evaluated whether AI assistance improved diagnostic accuracy or reduced turnaround times for ICH-positive cases. Results showed no significant difference in accuracy between radiologists with AI (99.2%) and without AI (99.5%). Sensitivity was slightly higher with AI (98.9% vs. 98.6%), but specificity was lower (99.3% vs. 99.8%). AI also did not significantly reduce report turnaround time (149.9 minutes vs. 147.1 minutes).
In conclusion, the AI triage tool did not improve radiologists' diagnostic performance or report times for ICH detection on head NCCT examinations. The lack of benefit may be due to the use of a widget without worklist reprioritization and the high diagnostic accuracy of experienced radiologists. Future research should focus on optimizing AI integration by implementing worklist reprioritization and assessing its effectiveness in diverse workflows, including those without 24-7 coverage, to better understand where AI can offer the most value.
Read full study
Prospective Evaluation of Artificial Intelligence Triage of Intracranial Hemorrhage on Noncontrast Head CT Examinations
American Journal of Roentgenology (AJR), 2024
Abstract
Background
Retrospective studies evaluating artificial intelligence (AI) algorithms for intracranial hemorrhage (ICH) detection on noncontrast CT (NCCT) have shown promising results but lack prospective validation.
Objective
To evaluate the impact on radiologists' real-world aggregate performance for ICH detection and report turnaround times for ICH-positive examinations of a radiology department's implementation of an AI triage and notification system for ICH detection on head NCCT examinations.
Methods
This prospective single-center study included adult patients who underwent head NCCT examinations from May 12, 2021 to June 30, 2021 (phase 1) or September 30, 2021 to December 4, 2021 (phase 2). Before phase 1, the radiology department implemented a commercial AI triage system for ICH detection that processed head NCCT examinations and notified radiologists of positive results through a widget with a floating pop-up display. Examinations were interpreted by neuroradiologists or emergency radiologists, who evaluated examinations without and with AI assistance in phase 1 and phase 2, respectively. A panel of radiologists conducted a review process for all examinations with discordance between the radiology report and AI and a subset of remaining examinations, to establish the reference standard. Diagnostic performance and report turnaround times were compared using Pearson chi-square test and Wilcoxon rank-sum test, respectively. Bonferroni correction was used to account for five diagnostic performance metrics (adjusted significance threshold, .01 [α=.05/5]).
Results
A total of 9954 examinations from 7371 patients (mean age, 54.8±19.8 years; 3773 female, 3598 male) were included. In phases 1 and 2, 19.8% (735/3716) and 21.9% (1368/6238) of examinations, respectively, were positive for ICH (P=.01). Radiologists without versus with AI showed no significant difference in accuracy (99.5% vs 99.2%), sensitivity (98.6% vs 98.9%), PPV (99.0% vs 99.7%), or NPV (99.7% vs 99.7%) (all P>.01); specificity was higher for radiologists without than with AI (99.8% vs 99.3%, respectively, P=.004). Mean report turnaround time for ICH-positive examinations was 147.1 minutes without AI versus 149.9 minutes with AI (P=.11).
Conclusion
An AI triage system for ICH detection did not improve radiologists' diagnostic performance or report turnaround times.